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Since  the  early  1950s  the  U.  S.  Air  Force  has  sponsored  the 
development  of  maintenance  simulators,  especially  those  which  could 
be  used  to  teach  the  troubleshooting  of  electronic  equipment..  Despite 
numerous  studies  which  demonstrated  their  cost  effectiveness,  the  use 
of  simulators  for  maintenance  training  still  is  not  widespread  and 
there  is  considerable  resistance  to  such  usage.  As  the  result  of 
recent  advances  in  simulation  technology,  and  the  need  to  conserve 
training  dollars,  the  Air  Force  is  making  a concerted  effort  to 
develop  more  information  about  maintenance  simulators  and  to  demon- 
strate their  applicability.  This  report  reviewed  past  and  present 
applications  of  simulation  to  maintenance  training,  concentrating  on 
the  najor  issues  and  problems  involved.  Also,  the  report  identified 
some  of  the  areas  where  further  research  data  is  required. 


Methodology 

The  review  concentrated  c,<  recent  1 iterature— since  1966— and 
made  extensive  use  of  secondary  sources  when  reviewing  the  earlier 
literature.  With  a few  exceptions,  the  articles  reviewed  pertained  to 
maintenance  training  and  the  use  of  simulators  and  low-cost  training 
devices  such  as  mock-ups.  Descriptions  on  research  methodology  and 
findings  were  kept  to  a minimum;  the  emphasis  was  on  identifying 
issues,  problems  and  areas  for  further  research  which  had  been  dis- 
cussed by  authors  of  recent  reports. 

Review  Topics 

Following  a historical  overview  of  the  research  on  maintenance 
simulators  and  related  training  devices,  the  report  concentrated  on 
reviewing  issues,  problems  and  research  findings  and  requirements  in 
five  areas.  Under  "Application  of  Simulation  Technology  to  Technical 
Training"  research  findings  relating  to  the  cost-effectiveness  of 
electronic  troubleshooting  trainers  and  low-cost,  low  fidelity  aids 
and  mock-ups  were  reviewed.  A section  dealing  with  the  "Determination 
of  Simulation  Requirements"  surveyed  current  procedures  and  problems 
relating  to  the  identification  of  tasks  to  be  simulated,  the  selection 
of  simulation  requirements,  and  the  relationship  between  simulation 
requirements,  learning  stages,  and  transfer  of  training.  Practices 
and  problems  concerning  the  "Design  and  Specification  of  Simulation 
Requirements"  were  discussed  in  terms  of  the  functional/physical 
fidelity  issue,  the  physical  and  instructional  features  which  might  be 
incorporated  into  a simulator,  and  the  procedures  for  selecting 
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simulator  requirements  and  developing  functional  specifications. 
Techniques  and  research  findings  relating  to  the  problem  of  obtaining 
instructor  and  student  acceptance  were  reviewed  under  "User  Acceptance 
of  Maintenance  Simulators."  Finally,  the  "Cost-Effectiveness  of  Main- 
tenance Simulators"  was  discussed  in  terms  of  the  conditions  which 
must  prevail  if  the  potential  of  maintenance  simulators  is  to  be 
realized. 

Conclusions 


The  reviewers  noted  that  current  issues  and  problems  related  to 
maintenance  simulators  are  similar  to  those  of  20  years  ago,  and  that 
many  questions  about  the  use  of  and  effectiveness  of  maintenance 
simulators  still  remain.  This  situation  exists  even  though  simulation 
technology  has  made  considerable  strides  in  recent  years,  due  in  large 
part  to  advances  in  computer  technology.  To  promote  further  the  use 
of  maintenance  simulators  the  authors  felt  that  research  and  develop- 
mental studies  in  the  following  areas  were  required: 

a.  Large  scale  field  studies  to  demonstrate  whether  or  not 
training  based  on  maintenance  simulators  actually  does 
transfer  to  the  field. 

b.  Development  of  exemplary  maintenance  programs  for  classes 
of  major  equipments.  These  programs  would  combine  the 
best  features  of  different  training  approaches--new 
technical  publications,  CAI,  maintenance  simu!ators--and 
would  attempt  to  provide  the  user  community  with  the 
experience  and  data  needed  to  commit  themselves  to  an 
investment  in  the  new  methods. 

c.  Development  of  exemplary  maintenance  simulators  for  major 
classes  of  equipment.  This  effort  would  be  designed  to 
provide  the  data  and  experience  needed  to  justify  making 
a decision  early  in  the  weapons  system  acquisition  cycle 
to  use  maintenance  simulators  instead  of  actual  equipment 
trainers. 

d.  Continued  investigation  of  procedures  to  identify  training 
requirements  which  can  be  supported  by  simulators.  This 
effort  should  concentrate  on  ways  to  identify  these  require- 
ments during  the  new  weapons  system  acquisition  cycle. 

e.  Continued  investigation  of  various  procedures  which  can  be 
used  to  develop  the  functional  specifications  for  main- 
tenance simulators,  to  include  both  the  tasks  which  should 
be  simulated  and  the  instructional  features  which  should 
be  incorporated  into  the  trainer. 
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Continued  exploration  of  techniques  for  obtaining  user 
acceptance,  especially  the  acceptance  of  instructors  and 
the  key  administrative  personnel  who  must  make  the 
decision  to  purchase  simulators  of  varying  degrees  of 
fidelity  as  opposed  to  actual  equipment. 

Comparative  evaluation  of  the  major  types  of  simulators— 
flat-panel,  three-dimensional,  and  computer-based— to 
determine  what  if  any  advantage  one  has  over  the  others. 

Development  of  improved  regulations  for  specifying  the 
points  in  the  new  weapons  acquisition  cycle  at  which 
decisions  about  simulators  should  be  made,  to  include 
who  the  decision-makers  and  the  resource  groups  should 
be,  and  what  improved  guidance  should  be  provided  to 
these  persons. 
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SIMULATORS  FOR  MAINTENANCE  TRAINING:  SOME  ISSUES,  PROBLEMS 
AND  AREAS  FOR  FUTURE  RESEARCH 

INTRODUCTION 


Review  Objectives 

Since  World  War  II  the  military  services,  and  especially  the 
Air  Force,  have  sponsored  considerable  research  on  the  development 
and  application  of  simulators  for  maintenance  training.  In  recent 
years  important  new  developments  have  occurred  in  this  area.  This 
report  will  review  past  and  present  applications  of  simulation  to 
maintenance  training  and  the  major  issues  involved  with  this  appli- 
cation. The  report  also  will  suggest  areas  for  future  research. 

Countless  research  studies,  field  tests  and  application  reports 
have  demonstrated  that  for  a variety  of  training  situations  simula- 
tors are  more  cost  effective  than  Actual  Equipment  Trainers  (AET). 
These  findings  are  fully  accepted  with  respect  to  pilot  training  (49); 
it  is  now  unthinkable  to  develop  a new  aircraft  without  also  develop- 
ing flight  simulators  for  pilot  training.  In  fact,  many  airlines 
believe  that  in  the  1980s  it  will  be  possible  to  conduct  total  pilot 
training  in  the  flight  simulator  (30).  This  attitude  towards  the  use 
of  simulators  does  not  yet  apply  to  the  training  of  maintenance 
personnel.  Despite  evidence  to  the  contrary,  there  still  is  a reluc- 
tance to  employ  maintenance  simulators;  a tradition  of  usage  has  still 
to  be  established.  Seme  of  the  reasons  for  this  will  be  examined  in 
this  report. 

Training  technologists  believe  that  maintenance  training  simula- 
tors should  be  employed  because  in  many  instances  they  are  less  costly 
and  more  effective  than  Actual  Equipment  Trainers  or  the  use  of  opera- 
tional equipment  for  training.  Research  and  application  studies  also 
have  shown  that  low-cost  training  equipments  and  training  aids  can  be 
cost-effective  substitutes  for  actual  equipment  trainers  or  expensive 
simulators.  A simulator  is  another  piece  of  equipnent,  and  one  should 
not  be  surprised  by  the  findings  that  it  too  can  be  replaced,  on 
occasion,  by  less  expensive  training  devices.  Selected  research  evi- 
dence in  support  of  this  "replacement"  will  be  cited. 

Review  Limitations 


While  preparing  this  report,  the  authors  did  not  attempt  to  ex- 
haustively search  through  all  possible  sources  of  literature  on  main- 
tenance training  and  maintenance  simulators.  Rather,  they  concentrated 
on  reviewing  the  recent  literature,  and  depended  on  secondary  sources 
whenever  possible.  Extensive  use  was  made  of  the  excellent  annotated 
bibliography  prepared  by  Valverde  (68).  That  document  summarizes  much 
of  the  research  literature  on  maintenance  training  up  through  1966. 
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Two  other  heavily  used  sources  wt  e the  Kinkade  and  Wheaton  chapter  on 
Training  Device  Design  (33)  and  G.  6.  Miller's  report  (43),  titled 
"Some  Considerations  in  the  Design  and  Utilization  of  Simulators  for 
Technical  Training  . " Miller,  Kinkade  and  Wheaton,  and  others  have 
summarized  the  research  evidence  related  to  maintenance  simulators, 
and  this  evidence  strongly  supports  the  contention  that  simulators  for 
maintenance  training  should  he  used  much  more  extensively.  Therefore, 
instead  of  reciting  much  of  the  research  evidence,  this  report  reviews 
issues,  problems,  and  research  requirements  related  to  the  application 
of  and  acceptance  of  the  research  evidence. 

Definitions 


Throughout  this  report  references  will  be  made  to  training 
devices,  training  aids,  simulators,  and  so  on.  Definitions  for  these 
terms  have  been  reviewed  by  Kinkade  and  Wheaton  (33).  Although  some- 
what loosely  formulated,  the  classification  of  training  devices  as 
shown  in  Figure  1 does  have  considerable  practical  value.  The  term 
"training  device"  is  used  to  encompass  all  training  devices  and  aids-- 
"any  arrangement  of  equipment  components,  apparatus,  or  materials 
which  provide  conditions  that  help  trainees  learn  a task."  In  turn, 
training  devices  are  divided  Into  two  classes,  training  aids,  and 
training  equipment.  Training  aids  are  defined  as  "devices  used  by  an 
Instructor  to  help  him  present  subject  matter."  Training  equipment 
Is  defined  as  "devices  which  provide  for  some  form  of  an  active 
practice  by  the  trainee." 


TRAINING  AIDS 
EXAMPLES 
FILMS 
TV 

GRAPHIC  MAT'L 
ETC 


TRAINING  OEVICES 


TRAINING  EQUIPMENT 


BABIJm 
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FLIGHT  NAVIGATION  &M. 
DRIVER  TRAINERS 
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Figure  1.  Classes  of  Training  Devices  (Adapted  from 
Kinkade  and  Wheaton,  1972) 


It  is  convenient  to  subdivide  training  equipment  into  "part-task" 
and  "whole-task"  trainers.  A part-task  trainer  gives  the  student  an 
opportunity  to  practice  selected  portions  of  a total  job  sequence;  the 
training  situation  does  not  require  total  work-context  inputs  or  total 
work-context  outputs,  just  those  related  to  a particular  task.  Part- 
task  trainers  are  not  intended  to  carry  a student  all  the  way  to  opera- 
ional  proficiency  on  the  task  being  trained  (46).  In  contrast,  a 
"whole-task"  trainer  by  definition  must  simulate  all  or  at  least  a 
major  segment  of  work-context  inputs  and  must  allow  for  the  making  of 
all  responses  appropriate  to  these  stimulus  inputs.  Whole-task  or 
total  task  training  can  occur  only  in  the  operational  situation  or  in 
a complete  simulator. 

Training  equipments  sometimes  are  referred  to  as  "trainers"  and 
at  other  times  as  "simulators."  Often  times  there  is  no  clear  dis- 
tinction between  a trainer  and  a simulator.  In  general,  the  term 
"trainer"  is  applied  to  training  devices  which  either  make  no  attempt 
to  realistically  represent  the  physical  appearance  of  equipment,  or, 
which  employ  all  or  portions  of  the  equipment  itself  for  training, 
e.g.,  actual  equipment  trainers.  On  the  other  hand,  a "simulator" 
attempts  to  faithfully  represent  the  stimulus  and  response  options 
provided  by  all  or  portions  of  a piece  of  equipment;  it  attempts  to 
provide  functional  fidelity.  The  simulator  also  may  attempt  to 
accurately  represent  the  appearance  and  "feel"  of  equipment.  This  type 
of  fidelity  is  called  either  equipment  or  physical  fidelity.  Obviously, 
simulators  can  have  varying  degrees  of  fidelity;  they  can  have  high 
physical  fidelity  and  somewhat  lower  functional  fidelity.  More  often 
they  have  high  functional  fidelity  and  somewhat  lower  physical  or 
equipment  fidelity.  What  difference  this  makes  with  respect  to 
training  will  be  a topic  for  later  discussion.  The  reviewers  noted  a 
lack  of  published  literature  concerning  the  cost-effectiveness  of 
recently  designed  maintenance  training  simulators,  and  they  expressed 
a hope  that  those  involved  with  maintenance  simulator  studies  will 
publish  their  findings  so  that  the  presumed  advantages  of  simulators 
can  be  adequately  documented. 
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SIMULATION  TECHNOLOGY  AS  APPLIED  TO  TECHNICAL  TRAINING 


Why  Simulate? 

Maintenance  personnel  are  trained  to  work  on  real  equipment.  Why 
then  not  use  real  equipment  for  maintenance  training?  In  the  first  in- 
stance, real  equipment  is  designed  to  serve  an  operational  need  not  re- 
lated to  training.  Therefore,  real  equipment  does  not  contain  the  de- 
sired features  of  a training  device.  Secondly,  real  equipment  is  al- 
most always  more  costly  than  a representative  training  device.  Finally, 
operational  requirements  take  precedence  over  training  requirements. 
Therefore,  training  centers  often  have  difficulty  getting  equipment 
and  spare  parts  because  these  are  sent  to  the  field. 

Miller  (42)  has  reviewed  the  reasons  why  the  Air  Force  is  inter- 
ested in  simulators.  In  addition  to  reducing  costs,  the  Air  Force 
feels  that  simulators  are  more  reliable  than  actual  equipment  trainers, 
allow  for  conduct  of  training  in  a safer  environment,  provide  a much 
greater  capability  for  malfunction  insertion,  and  are  much  less  noisy. 
These  also  are  the  reasons  why  the  Air  Force  and  the  airlines  are 
placing  increased  emphasis  on  the  use  of  flight  simulators  (28). 

Six  advantages  to  simulators  have  been  offered  by  Baker  and  War- 
nick  (5).  They  point  out  that  simulators* 

a.  provide  augmented  feedback  (extra  knowledge  of  results 
information) ; 

b.  increase  the  number  of  and  frequency  of  crises,  conflicts, 
breakdowns  and  emergencies  which  can  be  covered  during 
training; 

c.  reduce  the  operation  time  for  certain  events,  thus  increas- 
ing the  amount  of  practice  time; 

d.  allow  for  concentrated  practice  on  part-tasks,  and  for 
varying  the  sequence  of  part- task  training; 

e.  provide  guidance  and  stimulus  support  during  early  or 
initial  stages  of  learning;  and 

f.  allow  the  instructor  to  vary  the  difficulty  level  of  the 
task  to  be  learned,  and  allow  for  the  presentation  of  easy 
to  difficult  problems. 
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A Historical  Overview  of  Maintenance  Simulators 


Most  maintenance  simulators,  especially  those  for  electronic/ 
electrical  systems  and  subsystems,  have  been  designed  to  teach  the  con- 
ceptual aspects  of  organizational  or  between-stage  troubleshooting,  and 
the  interpretation  of  faulty  equipment  operation,  displays  and  test 
point  readings.  To  a lesser  extent  they  have  been  used  to  teach  within- 
stage  or  intermediate  level  troubleshooting.  They  have  been  designed 
this  way  because  of  the  general  belief  that  maintenance  training  should 
entail  ''instruction  in  decision  processes  to  a greater  degree  than  op- 
erations training.  This  instruction  must  include  formulation  of  deci- 
sion rules,  identification  of  decision  alternatives,  and  actual  decision 
making  (37)." 

Maintenance  simulators  also  can  be  used  to  teach  the  names  and 
locations  of  front  panel  displays  and  controls,  the  purpose  of  front 
panel  controls,  and  the  rudiments  of  how  to  energize  and  operate  the 
equipment.  Occasional ly,  simulators  are  employed  to  teach  preventive 
maintenance,  the  performance  of  system  self-checks,  and  the  interpre- 
tation of  normal  vs.  malfunction  operations.  In  addition  to  simulators, 
activated  mock-ups  may  be  used  to  teach  adjustments,  alignments,  and 
preventive  maintenance  routines. 

Valverde  (68)  has  reviewed  the  long  list  of  trainers  and  simula- 
tors supported  by  the  Air  Force.  Most  of  these  early  trainers  were 
two-dimensional;  a subsystem,  the  electrical  system  of  a weapons  system 
for  example,  was  represented  by  graphical  or  plug-in  "black  boxes", 
each  representing  a stage  of  the  equipment.  Test  points  were  provided 
on  a block  diagram  display  and  meters  might  be  used  to  show  symptom 
information.  Front  panel  controls  often  were  represented  by  pictures 
or  drawings,  or  by  switches.  Following  the  presentation  of  symptom 
information,  a student  was  required  to  select  the  logical  way  to  locate 
the  malfunctioning  black  box.  In  early  trainers,  test  point  informa- 
tion often  was  displayed  by  go/no-go  lights.  In  spite  of  their  crude- 
ness, these  devices  were  effective  in  that  they  could  be  used  to  teach 
students  the  logical  or  conceptual  aspects  of  troubleshooting.  This 
skill  is  taught  with  considerable  difficulty  using  real  equipment,  be- 
cause of  the  danger  in  and  tie  diffic  'n  inserting  malfunctioning 
parts  into  the  equipment. 

Along  with  two-dimensional  troubles. ^ - ...g  logic  trainers,  the  Air 
Force  also  sponsored  the  development  of  a few  three-dimensional  trainers, 
the  RMP-100A  Radar  Maintenance  Trainer  beinq  one  such  example.  In  the 
early  1960s,  Shriver,  Fink,  and  Trexler  (61)  developed  a trainer  which  con- 
verted a two-dimensional  troubleshooting  logic  trainer  into  a full-scale, 
three-dimensional  simulation  of  one  cabinet  of  the  NIKE  HIPAR  system. 

Test  points  for  troubleshooting  were  located  on  simulated  plywood 
chassis  contained  within  the  cabinet.  The  good  and  bad  waveform  outputs 
from  these  test  points  were  displayed  in  two  ways  on  successive 
iterations  of  trainer.  First  they  appeared  full  scale  on  a single 
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simulated  oscilloscope.  In  the  second  model  they  appeared  1/5  scale 
at  random  locations  on  an  egg  crate-like  display  matrix.  From  the 
students'  standpoint  it  made  no  difference  how  the  waveforms  were  dis- 
played even  though  the  second  display  method  was  very  low  in  physical 
fidelity  (60). 

Most  of  the  foregoing  trainers  were  manually  operated  in  that  each 
test  point  was  associated  with  a switch  on  an  instructor  console.  To 
set  up  for  a particular  malfunction  the  instructor  had  to  throw  a com- 
bination of  switches. 

In  the  1950s  and  1960s  a few  troubleshooting  trainers  were  devel- 
oped for  non-electronlc/non-electrical  systems.  The  tank  turret  mock- 
up  (17)  was  one  such  example.  Also,  during  that  period,  many  opera- 
tional mock-ups  were  constructed  to  demonstrate  system  operation  and  to 
provide  operator  training.  A few  elaborate  computer-driven  animated 
display  boards  also  were  developed.  These  devices  were  sometimes  used 
to  show  the  effects  of  a malfunction  on  system  operation,  but  the 
student  could  not  Interact  with  them  for  the  purpose  of  obtaining 
"hands-on"  experience  at  fault  isolation. 

As  digital  computers  became  more  powerful,  and  especially  smaller 
in  size,  the  feasibility  of  employing  them  to  control  small  trouble- 
shooting logic  trainers  became  evident.  Actually,  this  development 
began  in  the  late  1950s,  and  began  to  catch  on  in  the  mid-1960s  with 
the  development  of  the  SMART  trainer.  It  was  the  forerunner  to  the 
EC-I I/EC-3  trainers  currently  manufactured  by  Educational  Computer 
Corporation.  The  EC-I I/EC-3  trainers  consist  of  a large  display  panel 
containing  diagrams  and  pictures  of  all  the  major  components  of  a 
particular  equipment  subsystem,  and  all  the  "front  panel"  dials, 
switches,  and  meters  which  are  important  for  performing  an  operational 
checkout  of  the  subsystem.  The  panel  contains  numerous  test  points 
located  on  or  near  subsystem  components.  A small  computer  controls 
the  signal  supplied  to  each  test  point,  dial,  etc.,  in  such  a way  that 
the  normal  operating  system  can  be  simulated,  or,  one  of  many  malfunc- 
tions can  be  simulated.  An  instructor  console  is  used  to  insert  mal- 
functions into  the  trainer;  a viewing  console  is  provided  to  display 
for  the  trainee  what  he  would  see  on  his  meter  when  testing  at  a par- 
ticular point  or  when  viewing  certain  components. 

The  EC-I I/EC-3  type  trainers  are  not  used  to  teach  or  to  provide 
practice  on  perceptual -motor  skills.  However,  in  addition  to  concep- 
tual troubleshooting  skills,  they  can  be  used  to  teach  names  and  parts’ 
locations,  and  the  relation  between  operational  controls  and  equipment 
operation  (65). 

This  type  of  trainer  is  considered  to  be  "general  purpose"  be- 
cause its  mainframe  can  be  used  to  simulate  different  systems.  A 
simulation  package  for  any  particular  system  consists  of  a display 
panel,  a visual  disk  and  a computer  program  on  tape.  Given  that 


6 


two  or  more  simulation  packages  are  available,  it  takes  a very  short 
period  of  time  to  substitute  one  package  for  aiother. 

For  a variety  of  reasons  the  EC- I I/EC-3  trainers  are  corsidered 
to  be  an  advance  over  earlier  general  purpose  or  "generalized  trainers." 
Because  of  advances  in  miniaturization  it  now  is  easier  to  develop  a 
display  panel  which  represents  more  realistically  a major  equipment 
system.  Once  the  computer  program  fcr  a system  has  been  prepared, 
setting  up  the  trainer  to  simulate  c different  malfunction  can  be  done 
by  keying  in  a number  which  represents  the  new  malfunction.  The 
availability  of  compact  random  access  visual  systems  allows  for  the 
rapid  display  of  a wide  variety  of  test  signals,  pictures  of  equipment 
components,  and  so  on  at  a single  display  point.  As  mentioned  above, 
once  two  or  more  simulation  packages  have  been  developed,  it  is  easy 
to  substitute  one  for  the  other. 

The  extent  to  which  the  EC-I I/EC-3  trainer,  and  similar  flat- 
panel  trainers,  should  be  called  "general  purpose"  is  a matter  of 
definition.  Perhaps  it  would  be  more  accurate  to  describe  them  as 
having  a general  purpose  mainframe  which  employs  "systems  specific" 
simulation  packages.  The  flexibility  and  modifiability  of  these  pack- 
ages has  not  been  described  in  the  literature.  One  would  like  to  know 
how  easy  it  is  to  change  each  segment  of  the  simulation  package- 
computer  program,  visual  disk,  and  display  panel.  Weapons  systems  are 
subject  to  many  changes  during  their  lifetime  and  simulations  of  them 
also  should  be  capable  of  ready  modification. 

At  this  juncture  there  remain  many  unanswered  questions  about 
computer-driven  flat-panel  simulators.  They  may  have  a greater  poten- 
tial than  their  forerunners,  but,  in  terms  of  their  malfunction  simula- 
tion capacity,  their  capability  to  be  reconfigured  to  reflect  system 
modifications,  and  their  overall  effectiveness,  they  may  not  be  much 
of  an  improvement  over  their  forerunners.  In  fact,  their  forerunners 
may  have  had  certain  advantages.  For  example,  the  forerunners  may 
have  been  more  reliable;  at  least  some  of  them  were  easier  to  repro- 
gram or  reconfigure,  and  some  of  them  could  simulate  a very  wide  variety 
of  malfunctions,  at  least  at  the  between-stage  level.  Moreover,  earlier 
versions  of  flat-panel  simulators  were  much  more  "general  purpose,"  but 
this  feature  was  obtained  at  the  price  of  physical  fidelity.  It  Is 
ironic,  but  it  appears  that  recent  efforts  to  develop  improved  trouble- 
shooting trainers  have  in  fact  led  away  from  the  general -purpose-1  Ike 
trainers  of  the  past,  e.g.,  FORECAST  trainer  (21)  and  Generalized 
Electronic  Troubleshooting  Trainer  (70),  to  trainers  which  are  quite 
systems  specific  or  at  least  use  very  specific  simulation  packages. 

The  EC-I I/EC-3  is  a two-dimensional  trainer,  and  many  persons  like 
the  greater  physical  fidelity  provided  by  three-dimensional  trainers. 
Both  the  Educational  Computer  Corporation  and  Honeywell  have  developed 
three-dimensional  trainers  which  are  programmed  by  mini -computers. 

These  devices  are  full-scale  mock-ups  of  real  equipment,  but  portions  of 
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the  equipment  which  are  not  essential  for  the  troubleshooting  problems 
selected  may  be  simulated  by  etched  front-panel  graphics,  non- 
operational  hardware,  inexpensive  plastic  forms,  and  so  on.  If  the 
equipment  being  represented  contains  many  similar  panels,  then  only  a 
few  panels  need  be  functionally  operational  while  the  remainder  can  be 
simulated  in  some  inexpensive  fashion  (44). 

Many  of  the  early  troubleshooting  trainers,  like  the  Optimal  Se- 
quence Trainer  and  the  Malfunction  Information  Trainer  (68),  were 
essentially  devices  for  presenting  paper-and-pencil  troubleshooting 
exercises.  In  fact,  most,  if  not  all,  of  the  problems  simulated  on  an 
EC-II  trainer  could  be  presented  using  these  simpler  trainers,  but  with 
less  physical  fidelity.  However,  in  comparison  with  their  relatively 
simple  predecessors,  it  is  much  easier  for  instructors  to  set  up  EC-II- 
like  trainers  to  represent  different  malfunctions--the  instructor  keys 
into  the  trainer  a number  representing  a particular  malfunction  and 
then  the  trainer's  computer  software  reconfigures  signal  generation  and 
display  devices  to  represent  that  malfunction. 

Graphical  material  can  be  employed  to  teach  certain  aspects  of  the 
troubleshooting  process.  It  follows  from  this  that  certain  of  the  con- 
ceptual aspects  of  troubleshooting  can  be  simulated  using  computer- 
generated graphical  displays  coupled  with  a random  access  slide  projec- 
tor. One  example  of  this  type  of  maintenance  simulator  is  the  computer- 
based  training  system  (55)  developed  by  General  Electric.  The  PLATO 
system,  developed  by  the  University  of  Illinois  in  conjunction  with 
Control  Data  Corporation,  also  has  the  capability  to  use  computer- 
generated graphics  to  simulate  equipment  and  to  present  maintenance 
problems. 

Technically  speaking,  we  have  arrived  at  a point  where  the  concep- 
tual aspects  of  maintenance,  especially  troubleshooting,  can  be  simu- 
lated on  paper  or  by  three  different  hardware  systems--f1at-panel  system 
simulators,  computer-graphics'  terminals,  or  three-dimensional  equipment 
simulators  (49).  The  cost  of  the  products  produced  by  these  four  ap- 
proaches varies  inversely  with  the  physical  fidelity  of  the  simulation 
produced  by  each  approach.  But,  aside  from  certain  equipment  character 
istics  or  maintenance  tasks  which  are  still  difficult  to  simulate,  it 
does  appear  that  the  state-of-the-art  in  simulation  technology  has  ad- 
vanced to  where  maintenance  simulators  can  be  used  to  produce  a per- 
formance capability  in  course  graduates  which  is  not  achievable  with 
traditional  actual  equipment  trainers.  Unfortunately,  the  need  for  or 
usefulness  of  this  increased  performance  capability  is  not  perceived  by 
many  instructors  nor  by  major  segments  of  the  training  system  or  train- 
ing program  acquisition  system.  This  seems  to  be  because  the  training 
system  seldom  has  been  expected  to  produce  personnel  with  a practical 
troubleshooting  capability  at  graduation  from  a basic  (3  level)  main- 
tenance course.  In  our  judgement  major  changes  in  capability  seldom 
come  about  through  a perceived  need.  Rather,  an  improved  technology 
creates  a need  for  more  of  the  technology.  As  the  cost-effectiveness 
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of  simulators  becomes  more  widely  known,  there  probably  will  be  in- 

rl'nfkfi  .Jg1Jation  for  producing  course  graduates  who  have  an  increased 
capability  to  troubleshoot  equipment. 


APPLICATION  OF  SIMULATION  TECHNOLOGY  TO  TECHNICAL  TRAINING: 

RESEARCH  FINDINGS' 


Electronic  Troubleshooting  Trainers:  Pre-1967 

In  1968  Valverde  (68)  published  an  annotated  bibliography  which 
reviewed  the  important  developments  and  research  related  to  maintenance 
training  from  1950  through  1966.  The  following  section  provides  a brief 
summary  of  the  research  reported  in  that  source. 

According  to  Valverde: 

Most  maintenance  trainers  were  developed  for  use  in 
the  electronic  career  field  because  of  the  need  to 
provide  pract  ce  in  troubleshooting.  Due  to  the 
greai.  expense  of  operational  equipment  and  time  re- 
quired to  provide  practice  in  troubleshooting, 
personnel  of  the  Maintenance  Laboratory,  Lowry  Air 
Force  Base,  undertook  the  development  of  prototype 
troubleshooting  trainers  more  than  a decade  ago. 

The  rationale  for  the  use  of  these  trainers,  in 
addition  to  economy  and  savings  in  time,  was  the 
fact  that  the  trainers  provided  trainees  more  ex- 
perience in  troubleshooting  than  they  could  receive 
under  normal  training  conditions. 

Unfortunately,  the  problems  which  generated  the  Air  Force's  inter- 
est in  simulators  in  the  early  1950s  still  persist,  and  the  above  quote 
is  equally  applicable  today. 

Most  cf  the  electronic  troubleshooting  trainers  developed  prior 
to  1967,  and  up  through  1977  also,  concentrated  on  providing  "a  realis- 
tically complex,  relative  low  cost,  reliable  substrate  on  which  the 
tasks  associated  with  electronic  maintenance  can  be  performed"  (47). 

Briggs  and  Duvall  (7)  demonstrated  that  the  E-4  Fire  Control  Trouble- 
shooting Trainer  was  an  effective  trainers.  He  noted  that  performance  on 
the  E-4  trainer  was  faster  than  on  an  actual  equipment  trainer  because 
time-consuming  checks  were  eliminated,  and  it  was  easier  for  the  instructor 
to  Insert  malfunctions.  These  two  characteristics  are  common  to  almost 
all  troubleshooting  trainers. 

In  a series  of  studies  using  the  MAC-1  and  MAC-2  trainers  (Malfunc- 
tion and  Circuitry  Trainer),  French  (23)  demonstrated  that  students 
trained  on  these  devices  did  just  as  well  on  a performance  test  as  did 
students  trained  on  bench  mock-ups  of  real  equipment.  Other  early  forms 
of  troubleshooting  trainers  which  have  been  effectively  employed  include 


10 


the  Malfunction  Information  Trainer  (14),  the  Optimal  Sequence  Trainer 
(9),  and  the  MTS-1,  Maintenance  Task  Simulator  (20). 

During  the  1950s  and  early  1960s  the  Air  Force  sponsored  the  devel- 
opment of  a general  purpose  trainer,  the  GETS  or  Generalized  Electronic 
Troubleshooting  Trainer  (70),  which  was  designed  to  teach  a general 
strategy  for  troubleshooting  eljctronic  equipment.  Another  trainer, 
the  Checkout  and  Maintenance  Trainer  (CAM)  (29),  was  developed  as  a re- 
search vehicle  for  studying  maintenance  training. 

From  1957  through  1965,  Shriver  and  his  colleagues  undertook  an 
extensive  series  of  studies  concerned  with  both  the  development  of  im- 
proved technical  documentation  and  maintenance  training  devices.  The 
beginning  study  in  this  series,  one  designed  to  improve  training  on  the 
M-33  radar  system,  employed  a mock-up  of  the  system  that  covered  about 
500  square  feet.  Using  specially  developed  analysis  techniques,  the 
radar  system  was  divided  conceptually  into  troubleshooting  blocks  such 
that  if  the  signal  into  a block  was  correct  but  one  or  more  of  the  block 
outputs  was  not,  then  the  fault  had  to  reside  in  one  or  more  of  the 
parts  which  comprised  the  block  (60,  62). 

The  training  device  developed  represented  the  system  at  the  trouble- 
shooting block  level.  Each  block  ./as  simulated  by  a small  metal  "black 
box"  named  for  the  troubleshooting  block  it  represented.  Each  "black 
box"  had  a test  point  which  was  wired  to  a large  upright  instructor 
panel.  The  IOC-plus  black  boxes  which  represented  the  entire  radar  sys- 
tem were  placed  on  racks.  A simulated  oscilloscope  was  used  to  display 
waveform  readings  at  each  test  point,  and  a simulated  malfunction  could 
be  placed  within  any  "black  box"  by  flicking  the  instructor  console 
switch  for  that  box.  If  the  reading  at  a test  point  was  supposed  to  be 
correct,  a glass  plate  containing  an  etching  of  a normal  reading  lit  up 
within  the  simulated  oscilloscope;  otherwise  a straight-line  "abnormal 
reading"  etching  was  displayed.  Students  used  this  trainer,  along  with 
a troubleshooting  block  diagram  of  the  system,  to  learn  how  to  trouble- 
shoot conceptually  to  the  small  group  of  parts  which  comprised  each 
troubleshooting  block.  Sometimes  the  troubleshooting  sequence  began 
with  the  stuaent  collecting  "symptoms"  information  on  an  inexpensive 
front-panel  mock-up  of  the  M-33.  At  other  times,  an  instructor  pro- 
vided the  initial  symptoms  (5s»). 

The  original  FORECAST  trainer  was  considered  to  be  a general  pur- 
pose or  generalized  trainer  in  that  it  was  fairly  easy  to  modify  the 
trainer  to  represent  another  system.  To  accomplish  this,  the  plates  in 
the  simulated  oscilloscope  and  the  names  of  the  "black  boxes"  had  to  be 
changed.  Of  course,  before  these  steps  could  b”  accomplished,  the  sys- 
tem to  be  simulated  had  to  be  analyzed  and  subdivided  into  trouble- 
shooting blocks.  This  step,  often  the  most  time-consuming  and  costly 
one  in  the  simulator  development  process,  has  to  occur  for  any  simu- 
lator which  has  been  configured  to  represent  a specific  system. 
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A second  version  of  the  trainer  just  described  was  used  to  teach 
maintenance  of  the  NIKE  HIPAR  system.  It  too  was  considered  to  be  a 
general  purpose  trainer  in  that  its  basic  features  could  be  employed 
to  simulate  any  electronic  system  once  a specially  prepared  block  dia- 
gram had  been  developed  for  the  system  (21,  61). 

In  this  trainer  the  numerous  racks  of  black  boxes  which  typified 
the  original  FORECAST  trainer  were  discarded;  they  were  replaced  by  a 
block  diagram  located  on  the  surface  of  a 2-  by  3- foot  metal  box  called  a 
"desk-top"  mock-up.  Test  points,  wired  to  toggle  switches  located  on 
o control  console,  came  out  on  each  square  inch  of  the  top  surface  of 
the  mock-up.  By  carefully  positioning  the  blocks  of  a block  diagram  it 
became  possible  to  place  the  diagram  on  top  of  the  mock-up  such  that 
there  was  a test  point  at  the  input  and  output  of  each  block.  As  an 
added  feature,  the  mock-up  unit  contained  rotary  and  toggle  switches 
which  represented  the  main  operational  controls  and  monitor  switches 
of  the  system.  On  this  mock-up  students  could  practice  the  cognitive 
aspects  of  troubleshooting--symptom  interpretation,  the  programming 
of  troubleshooting  checking  sequences,  and  the  interpretation  of  indi- 
vidual checkpoint  information. 

A third  unit  of  the  above  trainer  was  called  a Display  or  Wave- 
former  Unit.  It  consisted  of  a metal  matrix  of  156  cells  each  contain- 
ing a small  lamp.  Each  cell  was  associaJd  with  a particular  test  point 
on  the  mock-up  and  a particular  switch  the  control  console  unit. 

When  a particular  test  point  was  touched  with  a probe,  a lamp  was  lit 
in  the  associated  cell  of  the  Waveformer  Unit.  Over  the  front  surface 
of  the  Waveformer  Unit  was  placed  a photographic  negative  containing 
pictures  of  the  normal  waveform,  voltage  or  meter  readings  which  should 
appear  at  each  test  point.  All  incorrect  readings  were  shown  as  a 
straight  line  in  one  cell  of  the  Waveformer. 

In  many  respects  the  mock-up  just  described  had  less  physical  fidel- 
ity than  its  predecessor.  In  particular,  the  Waveformer  had  much  less 
physical  fidelity  than  the  simulated  oscilloscope  used  in  the  M-33  study. 
Despite  this,  the  two  trainers  seemed  to  be  equally  effective,  and  the 
low  fidelity  of  the  Waveformer  had  no  apparent  psychological  or  behav- 
ioral impact  on  students.  From  a practical  standpoint,  the  second 
trainer  was  as  effective  as  the  first  even  though  it  was  less  expensive, 
smaller  and  had  less  physical  fidelity. 

In  a third  version  of  the  foregoing  trainers  the  device  was  designed 
so  that  students  could  insert  their  own  malfunction  patterns  by  pressing 
keyboard  buttons.  A manual  was  produced  which  gave  the  initial  state- 
ments of  malfunctions  (previously  given  by  the  instructor)  and  the  ap- 
propriate keyboard  button  settings.  The  student  then  took  test  point 
readings  to  select  other  test  points  until  they  finally  had  localized 
the  malfunction  to  a troubleshooting  block  (63). 
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In  the  final  version  of  the  FORECAST  trainer  the  entire  trouble- 
shooting process  was  simulated  on  paper  and  incorporated  into  a trouble- 
shooting manual.  This  manual  contained  all  the  scope  pictures  that 
previously  were  displayed  on  the  simulated  oscilloscope,  all  the  symp- 
tom information  that  was  provided  by  an  instructor  or  on  "initial  state- 
ments of  malfunctions,"  as  well  as  other  information  such  as  location 
of  test  points.  In  addition,  troubleshooting  exercises  in  scrambled 
book  format  were  developed  (63). 

None  of  the  four  versions  of  FORECAST  trainers  was  compared  with 
another,  but  all  seemed  equally  effective.  All  versions  taught  con- 
ceptual troubleshooting  skills;  all  were  generalized  trainers  in  that 
they  could  be  reprogrammed  to  represent  other  electronic  systems. 


Electronic  Troubleshooting  Trainers:  Post-1966 

Flat-Panel  System  Simulators.  From  the  mid-1960s  to  the  present 
considerable  advances  have  been  made  in  tne  development  of  computer- 
driven  2-dimensional  troubleshooting  trainers.  Earlier  versions  of 
these  trainers  suffered  because  they  could  not  display  all  portions  of 
a system  relevant  to  a set  of  troubleshooting  problems,  could  not  show 
controls  graphically,  and  often  displayed  test  point  and  symptom  in- 
formation in  a binary  fashion.  Furthermore,  until  recently  even  so- 
called  general  purpose  trainers  could  not  be  changed  rapidly  from  one 
subsystem  to  another,  nor  could  they  easily  simulate  non-el ectronic 
subsystems.  Today  flat-panel  trainers  such  as  the  EC- I I /EC-3  trainers 
can  easily  be  changed  from  one  system  to  another  and  can  represent 
both  electronic  and  non-electronic  subsystems  provided  that  programs 
are  available  for  doing  so.  It  is  worth  remembering,  however,  that 
this  flexibility  is  obtained  by  substituting  one  complete  simulation 
package  for  another.  These  packages  are  costly  to  prepare,  and  the  ease 
with  which  the  panels  and  programs  which  comprise  the  package  can  them- 
selves be  modified  remains  to  be  determined. 

Numerous  researchers  have  investigated  the  effectiveness  of  the 
EC- 1 1 . McGuirk,  et  al  (39)  simulated  the  AN/APQ-126  radar  and  compared 
its  effectiveness  for  teaching  troubleshooting  with  the  effectiveness 
of  Actual  Equipment  Trainers  (AET).  They  found  that  the  simulator  was 
as  effective  as  the  AET,  was  more  reliable,  and  was  much  less  costly. 

Spangenberg  (65)  demonstrated  that  the  AN/APQ-126  simulator  could 
be  used  effectively  to  teach  national  guardsmen  the  purpose  of  controls, 
the  interpretation  of  normal  versus  malfunction  operation,  the  perform- 
ance of  system  self-checks,  and  malfunction  isolation  procedures. 

Malone,  et  al  (38)  reviewed  the  effectiveness,  usability  and  cost 
of  a variety  of  trainers,  and  concluded  that  the  EC- 1 1 should  receive 
top  ranking  as  a troubleshooting  skills  trainer. 
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A number  of  maintenance  courses  throughout  the  armed  services  employ 
maintenance  simulators,  the  EC-I I/EC-3  in  particular.  Apparently  they 
are  effective  but  there  is  little  published  research  evidence  to  con- 
firm this  supposition. 

Computer-Graphics  Terminals.  Computer-based  instruction  (CBI)  is 
used  effectively  in  a number  of  service  maintenance  courses.  In  addi- 
tion to  its  use  for  teaching  conceptual  material,  increasing  interest 
has  been  shown  in  the  use  of  CBI  for  teaching  maintenance  skills,  the 
use  of  test  equipment  (11)  being  one  such  example.  Stern  (66)  used 
computer-graphics  terminals  to  teach  the  use  of  the  oscilloscope.  He 
found  that  training  using  CBI  techniques  was  as  effective  as  using 
actual  oscilloscopes;  neither  training  was  very  effective  in  the  ab- 
solute sense,  however.  Lahey  (35)  demonstrated  that  CBI  can  be  used 
to  teach  the  use  of  the  multimeter.  However,  the  end-of-training  test 
was  verbal  rather  than  psychcmotor. 

In  a study  for  the  Navy,  Radsken  and  Grosson  (54)  concluded  that 
CBI  can  be  effective  in  the  submarine  systems  training  environment 
when  applied  to  interactive  operator  and  basic  maintenance  training 
tasks. 

At  the  Behavioral  Technology  Laboratories  at  the  University  of 
Southern  California,  the  use  of  generative  computer-assisted  instruction 
(CAI)  for  task  training  is  under  investigation  (57).  Two  stand-alone 
CAI  systems  are  being  developed  for  the  U.  S.  Navy;  one  system  to  teach 
maintenance  of  surface  ship  electronic  equipment,  and  the  other  to  teach 
maintenance  of  the  F-14  AWG-9  Weapons  Control  System.  For  both  projects 
computer  graphics  terminals  will  simulate  equipment  front-panel  con- 
figurations. Using  a light  pen  the  student  will  be  able  to  interact  with 
front-panel  controls  and  displays.  This  type  of  CAI  system  can  be  placed 
on  board  ship  and  eventually  should  have  the  capability  to  serve  both  as 
a training  device  and  a job  aid. 

In  a somewhat  similar  project  the  Air  Force  Human  Resources  Labora- 
tory has  developed  a "Computer-Assisted  Performance  Carrel."  This 
device  "which  consists  of  a computer  terminal  (PLATO  IV/Magnavox) , a 
digital  control  processor,  and  analog  interface  equipment,  incorporates 
many  of  the  standard  features  of  the  EC- I I plus  interactive  instruc- 
tional programming.  This  interactive  approach  provides  a self-paced 
capability  thereby  dramatically  reducing  the  monitoring  functions  in- 
structors normally  perform  in  performance  courses  (45)."  The  Perform- 
ance Carrel  is  designed  to  teach  "hands-on"  tasks;  its  effectiveness 
for  this  purpose  is  the  subject  of  current  studies. 

Three-Dimensional  Trainers.  Within  recent  years  a new  class  of 
three-dimensional  trainers  has  been  coveloped  by  a number  of  corpora- 
tions. Essentially  these  are  EC-II-like  simulators  which  have  been 
housed  in  a three-dimensional  mock-up  of  the  equipment  being  simulated. 
This  provides  an  added  dimension  of  physical  realism  which  probably 
will  increase  user  acceptance. 
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The  Educational  Computer  Corporation  has  developed  a 3-D  simulator 
for  the  Navy's  A-7E  Heads-Up  Display  Unit  and  Test  Bench.  Not  all  of 
the  test  bench  is  simulated,  but  its  external  appearance  is  represented 
very  realistically.  Honeywell  is  in  the  process  of  developing  a three- 
dimensional  simulator  of  the  Air  Force's  6883  Converter/Flight  Controls 
Test  Station  for  the  F-111D  aircraft.  To  date  no  research  evidence  has 
been  reported  on  the  effectiveness  of  the  above  two  simulators.  Al- 
though more  costly  than  two-dimensional  simulators  such  as  the  EC-3, 
there  appears  to  be  no  reason  why  they  should  not  be  equally  as  effec- 
tive. Whether  or  not  they  will  be  as  cost-effective  as  the  EC- II/ 

EC-3  is  a question  for  further  research  to  decide. 

Until  recently  simulators  have  been  used  to  teach  maintenance  at 
the  organizational  level  but  not  at  the  intermediate  level.  Now  with 
the  development  of  three-dimensional  simulators  for  test  bench  equip- 
ment, sophisticated  trainers  are  available  for  teaching  intermediate 
level  maintenance.  It  is  now  possible  to  develop  such  trainers  be- 
cause of  advances  in  the  design  of  electronic  equipment.  Recently  de- 
signed electronic  systems  tend  to  be  interlaced  with  sensors  connected 
to  front  panel  displays  and  built-in  test  equipment  (BITEl.  The  sen- 
sors monitor  the  function  of  removable  modules,  drawers,  card  racks, 
etc.  called  "line  replaceable  units"  (LRU).  At  the  organizational  level 
of  maintenance  the  BITE  and  the  front  panel  displays  are  used  to  isolate 
a malfunction  to  an  LRU.  Essentially  this  involves  the  performance  of 
a series  of  operator-like  actions.  Through  the  use  of  sensors  within 
LRUs,  and  the  use  of  signal  generators  controlled  by  a computer,  it 
has  become  possible  to  develop  test  sets  and  stations  which  can  isolate 
a malfunction  to  a small  section  of  an  LRU  or  even  to  an  individual  part. 
This  is  maintenance  at  the  intermediate  level  and  it  too  is  accomplished 
via  the  performance  of  operator-like  tasks.  The  implication  of  these 
developments  is  twofold.  Firstly,  troubleshooting  at  both  the  organiza- 
tion and  intermediate  levels  of  maintenance  has  become  quite  similar  in 
nature— they  tend  more  and  more  to  involve  operator- like  tasks.  Second- 
ly, it  is  easier  to  simulate  operator  tasks  than  other  maintenance  tasks 
such  as  the  removal  and  replacement  of  components.  Moreover,  it  is  quite 
acceptable  to  use  simulators  to  teach  equipment  operation.  Therefore, 
it  is  becoming  more  and  more  acceptable,  as  well  as  technically  easier, 
to  use  simulators  to  teach  fault  isolation  at  the  intermediate  mainten- 
ance level. 

A recent  report  by  Parker  (51)  "found  that  the  commonality  is  in- 
creasing among  avionics  systems  that  are  members  of  the  same  family  and 
among  equipments  that  are  members  of  different  families."  He  suggested 
that  there  will  be  a decline  in  skill  level  requirements  for  avionics 
technicians  because  of  the  increased  commonality  between  equipments 
and  because  of  the  increased  modularization,  integration  and  potting  of 
circuits.  While  investigating  the  use  of  CAI  maintenance  training 
systems  for  the  Navy,  Rigney  and  Towne  (57)  concluded  that  "trouble- 
shooting at  the  basic  circuit  level  seemed  likely,  as  a consequence  of 
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medium  and  large  scale  integration  of  circuits,  to  become  less  impor- 
tant." As  a consequence  they  concluded  also  that  troubleshooting  on  the 
basis  of  front  panel  symptoms  will  become  more  important. 

Troubleshooting  Trainers:  Non-Electronic  Systems 

As  mentioned  earlier,  most  troubleshooting  trainers  have  been  de- 
signed to  simulate  electronic  subsystems.  Numerous  researchers,  how- 
ever, have  demonstrated  that  mock-ups  of  front  panel  displays  and  con- 
trols of  radar  systems  (10,  61),  or  the  instvftnent  panel  and  driver's 
control  of  a tank  (17),  can  be  used  effectively  to  teach  nomenclature 
and  location,  and  energizing  and  shut-down  procedures.  These  trainers, 
however,  could  not  be  used  to  teach  troubleshooting  procedures  other 
than  the  performance  of  operational  checks  and  symptom  gathering.  More 
recently,  the  EC-II  has  been  used  to  successfully  teach  trouble- 
shooting of  hydraulic  systems  and  the  53C51  Mohawk  propeller  system 
(18).  As  part  of  a project  for  the  Navy,  Burtek  Corporation  has 
developed  a simulator  for  the  Hagan  Automatic  Boiler  Control  System. 

A simulator  somewhat  similar  to  the  EC-II,  the  Omnidata  simulator, 
is  used  by  the  Navy  to  teach  start-up  and  running  procedures  for  in- 
ternal combustion  engines.  Omnidata  simulators  are  fixed  purpose 
simulators;  they  lack  the  programmable  features  cf  the  EC-II.  They  can, 
however,  be  configured  to  simulate  a variety  of  equipments  and  systems 
to  include:  two-cycle  diesel  engines;  CTC-85;  Ingersol  Rand  Air  Com- 
pressor; Automotive  Lighting,  Indicating  and  Warning  Systems;  Aero-47- 
A Weapons  Loader;  GTC-100;  and  the  NR-10  Air  Conditioning  System.  The 
device  can  be  used  to  demonstrate  principles  of  operation  and  trouble- 
shooting procedures.  A few  Omnidata  models  can  be  purchased  a,  off- 
the-shelf  items,  but  most  models  must  be  custom  designed. 

Effectiveness  of  Low-Cost,  Low-Fidelity  Aids  and  Mock-Ups 

A number  of  studies  have  demonstrated  that  "very  simple,  low-cost 
trainers,  or  mock-ups  having  only  gross  physical  fidelity--dummy  instru- 
ment panels,  photos,  or  line  drawings--can  be  used  effectively  to  train 
students  to  perform  a variety  of  procedural  tasks  (68)."  In  1954  Denen- 
berg  (17)  showed  that  a very  inexpensive  mock-up  of  the  instrument  panel 
and  driver's  controls  of  the  M47  tank  was  as  effective  as  an  expensive 
mock-up  and  the  real  tank  for  teaching  starting  and  stopping  procedures, 
and,  the  names,  locations,  ard  functions  of  driver  instruments  and  con- 
trols. Cox,  et  al . (10)  were  unable  to  demonstrate  a relationship  between 
12  training  devices  of  varying  degrees  of  reduced  functional  and  physical 
fidelity  and  student  proficiency  or  training  time  on  fixed-procedure 
tasks. 

Torkelson  (67)  compared  the  effectiveness  of  cutaways,  mock-ups, 
and  transparencies  for  providing  instruction  on  the  Mark  13  torpedo. 

He  found  no  differences  in  training  effectiveness  of  the  three  kinds  of 
media  but  did  find  slightly  better  retention  when  three-dimensional 
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trainers  were  employed.  He  recommended  that  mock-ups  and  cutaways  be 
justified  on  grounds  other  than  training  effectiveness,  such  as  require- 
ment for  outdoor  use,  student  operation,  need  to  show  moving  parts,  and 
so  on. 

As  already  reviewed  in  this  report,  Shriver,  Fink,  and  Trexler  found 
that  the  conceptual  skills  of  torubleshooting  could  be  taught  equally  well 
using  scrambled  book  exercises  or  a variety  of  low-cost,  low  fidelity 
trainers.  In  addition,  they  found  that  inexpensive  mock-ups  could  be 
used  to  teach  radar  operating  procedures  (60). 


DETERMINATION  OF  SIMULATION  REQUIREMENTS 


Current  Procedures  and  Problem  Areas 


The  general  procedures  for  determining  simulation  requirements  are 
quite  easily  stated.  One  first  conducts  a job  and  task  analysis  to 
identify  training  requirements.  These  requirements  then  are  examined 
to  identify  those  which  should  be  covered  during  training.  This  sub-set 
of  requirements  is  then  examined  in  terms  of  the  training  media  and 
methods  needed  to  support  training  on  each  requirement.  Eventually  a 
smaller  sub-set  of  training  requirements  is  identified  which  probably  can 
can  best  be  taught  through  the  use  of  a simulator.  Later  on,  during  a 
trade-off  analysis,  one  identifies  those  requirements  which  are  not 
feasible,  or  which  are  too  costly,  to  simulate.  A decision  is  made  to 
handle  these  requirements  by  some  other  means,  probably  OJT,  or  not  to 
handle  them  at  all.  Specifications  then  are  prepared  for  a simulator. 

By  turning  to  appropriate  reference  sources,  one  can  find  rather  elabor- 
ate descriptions  for  how  to  perform  each  of  the  foregoing  steps.  But, 
after  digesting  all  that  information,  most  practitioners  still  would 
agree  with  Bryan  and  Regan  (8)  that  there  are  few  if  any  prescriptions 
for  how  to  identify  simulation  requirements  and  to  design  major  training 
devices. 

Most  training  equipment  designers  can  agree  with  Modrick  (48)  that 
one  should  not  develop  specifications  for  training  content  and  training 
devices  until  training  objectives  have  been  identified.  Almost  any  docu- 
ment describing  the  Instructional  System  Development  process  will  em- 
phasize this  point.  All  efforts  to  design  training  programs  and  de- 
vices should  begin  with  a thorough  job  and  task  analysis.  However,  des- 
pite continual  advances  in  the  state-of-the-art,  techniques  for  accom- 
plishing a job  and  task  analysis  are  still  more  of  an  art  than  a science. 

To  further  compound  the  problem,  it  is  becoming  increasingly  obvious 
that  "job-descriptive  information  alone  cannot  provide  sou. id  basis  for 
translating  job  requirements  into  personnel  and  training  requirements  (8)." 

In  recent  reports,  both  Cream,  et  al.  (13)  and  Eggemeier  (19)  have 
discussed  the  need  to  develop  more  efficient  means  and  methods  for 
identifying  training  device  requirements.  Cream  reviewed  the  limitations 
of  current  methods  and  found  them:  not  sufficient  for  the  actual  de- 
sign of  a training  device;  difficult  to  apply  by  unskilled  persons;  and 
apt  to  emphasize  minimally  acceptable  performance.  Eggemeier  reported 
that  short  term  methods  for  identifying  training  device  requirements 
result  in  the  adequate  specification  of  training  requirements  but  do  not 
provide  satisfactory  information  upon  which  to  base  decisions  about  what 
features  the  training  device  should  possess.  Bryan  and  Regan  feel  that 
much  more  basic  research  is  required  to  "expand  and  validate  the 
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behavioral  information..."  provided  -by  job  ano  task  analysis.  They 
cite  also  the  need  for  more  information  on  the  relationship  between 
"various  kinds  of  tasks  and  various  kinds  of  learning." 

Current  procedures  for  determining  simulation  requirements  can, 
with  difficulty  and  often  unknown  effectiveness,  be  applied  to  existing 
weapons  systems.  They  are  much  more  difficult  to  apply  to  new  systems 
still  under  development.  Both  Modrick  (48)  and  Cream  (12)  have  express- 
ed the  need  for  methods  which  will  identify  training  requirements  and 
training  device  requirements  early  in  the  development  cycle  for  a new 
system.  Their  review  of  this  problem  reveals  that  we  do  not  have 
methodology  for  converting  engineering  data  into  training  and  training 
device  requirements.  This  problem  area  has  been  studied  periodically 
since  at  least  the  mid-1950s  (58,  59)  but  few  solutions  have  been 
forthcoming,  at  least  with  respect  to  maintenance  training. 

In  one  of  the  FORECAST  studies  (59)  Shriver  took  the  position  that 
changes  made  during  the  last  year  before  fielding  a new  electronic 
weapons  system  are  trivial  with  respect  to  the  level  of  detail  needed 
to  develop  a maintenance  simulator  which  can  teach  troubleshooting 
at  both  the  organizational  and  intermediate  levels  of  maintenance. 

They  therefore  designed  and  built  three  types  of  maintenance 
simulators  based  on  a detailed  task  analysis  of  the  NIKE  HIPAR  system 
as  it  existed  one  year  before  it  was  scheduled  to  become  operational. 

The  analysis  and  development  process  was  accomplished  within  one  year, 
beating  the  real  equipment  to  the  field  by  several  months.  The 
simulators  were  the  only  equipment  the  Ordnance  Guided  Missile  School 
had  to  train  on  for  six  months,  and  they  served  as  training  equipment 
for  several  years  after  the  real  equipment  became  available  at  the 
training  site. 

Both  Val verde  (68)  and  Reilly  (56)  have  discussed  problems  asso- 
ciated with  identifying  training  requirements  and  devices  for  new 
systems.  When  training  requirements  and  training  programs  are  under 
development  concurrently  or  in  advance  of  systems  in  the  field,  it 
often  is  necessary  to  make  a purchasing  decision  before  clearly  de- 
fined alternatives  in  the  form  of  simulators  or  other  training  media 
have  been  developed.  This  may  Ibrce  the  purchase  of  actual  equip- 
ment for  training.  It  is  easier  to  order  additional  equipment  from 
the  prime  contractor  than  it  is  to  order  training  equipment  from  a 
separate  vendor. 

Once  training  requirements  have  been  identified  there  still  is  a 
need  to  select  those  tasks  which  should  be  simulated.  Miller  (44) 
has  discussed  one  method  for  doing  this.  It  involves  deriving  a weight- 
ed rank  order  for  each  task  to  be  covered  during  training.  Ranks  are 
based  on  the  difficulty,  frequency  and  importance  of  the  task.  Those 
tasks  receiving  the  highest  composite  rank  are  given  proportionate 
consideration  regarding  whether  or  not  they  should  be  simulated,  and, 
the  level  of  fidelity  at  which  they  should  be  simulated.  This  is  a 
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practical  way  of  making  decisions  about  fidelity  and  about  which  tasks 
should  be  candidates  for  simulation.  Of  course,  once  decisions  have 
been  made  about  which  training  requirements  should  be  supported  by  a 
simulator,  one  still  has  to  make  a host  of  decisions  regarding  the 
feasibility  and  cost  of  designing  a simulator  to  accomplish  the  desired 
effect.  These  problems  will  be  discussed  in  the  next  section  of  this 
report. 

Transfer  of  Training,  Learning  Stages  and  Simulation  Requirements 

What  is  learned  earlier  in  a course  should  aid  learning  during 
latter  stages  of  the  course.  When  it  does,  positive  transfer  of  training 
is  said  to  have  occurred.  A well-designed  training  sequence  seeks  to 
control  the  training  environment  so  as  to  maximize  positive  transfer  8) 
from  one  portion  of  a course  to  another,  and  from  the  course  to  the  ^ob. 

Transfer  of  training  should  be  viewed  with  reference  to  stages  of 
learning.  As  shown  in  Table  I,  a novice  repairman  enters  a training 
program  the  general  goal  of  which  is  to  prepare  him  to  the  point  where 
he  can  attain  final  mastery  of  his  job  in  the  work  environment.  As  the 
student  encounters  each  new  task  during  his  formal  training,  he  pro- 
gresses through  three  stages  of  learning  with  respect  to  that  task. 

In  this  report  these  learning  stages  are  defined  as  being  very  similar 
to  the  three  stages  of  learning  described  by  R.  B.  Miller  (47).  How- 
ever, we  find  it  useful  to  confine  Miller's  three  stages  to  formal 
or  "school house"  learning,  and  to  recognize  that  a fourth  stage  of 
learning  or  practice  on  the  job  is  almost  always  required. 

Numerous  authors  (33,  43,  47)  have  suggested  that  different  kinds 
of  tasks  and  different  degrees  or  stages  of  learning  have  different 
implications  for  transfer  of  training.  We  still  do  not  know  enough 
about  these  relationships,  and  it  is  an  area  well  worth  further  research. 
In  particular,  we  need  more  research  on  how  to  determine  the  transfer 
validity  of  trainers  before  they  are  accepted  (1).  In  the  meanwhile, 
considerable  evidence  does  exist  to  support  the  view  that  training  re- 
quirements during  earlier  stages  of  learning  are  different  from  those 
of  later  stages,  and  therefore  require  support  from  devices  which  have 
different  characteristics.  For  example,  R.  B.  Miller  (47)  defined 
three  classes  of  trainers  --  familiarization  trainers,  constructed- 
response  trainers,  and  automatized  skill  trainers.  He  associated  each 
class  with  a stage  of  learning,  and  pointed  out  that  each  class  has  dif- 
ferent characteristics.  Kinkade  and  Wheaton  (33)  have  divided  the 
learning  spectrum  into  five  stages  and  have  associated  each  with  par- 
ticular types  of  training  aids  and  devices.  Collectively  the  research 
literature  suggests  that,  as  showi;  in  Table  II,  various  types  of  train- 
ing devices  and  media  can  best  be  employed  to  support  various  stages  of 
learning. 
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Table  I.  General  Relationship  Between  Stages  of  Learning 
Training  Sites,  and  Training  Objectives 
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Training  aids,  devices  and  simulators  are  employed  to  support  the 
learning  process.  Their  usage  also  should  be  viewed  in  terms  of  efforts 
to  increase  transfer  of  training.  According  to  this  point  of  view, 
training  aids  are  used  early  in  the  learning  process  so  as  to  decrease 
the  time  and  effort  required  to  acquire  skills  and  knowledges  later  on. 
Part-task  trainers  are  employed  so  that  trainees  need  spend  less  time  on 
job-segment  or  whole-task  trainers.  Job-segment  or  whole-task  trainers 
are  used  when  it  is  necessary  to  reduce  the  amount  of  time  required  by 
course  graduates  to  acquire  acceptable  on-the-job  proficiency.  When 
viewed  from  this  prospective,  it  becomes  apparent  that  no  one  class  of 
training  aids  or  devices  need  carry  the  entire  training  load.  Further- 
more, it  becomes  obvious  that  from  a cost-effectiveness  standpoint  the 
general  training  strategy  should  be  to  obtain  full  measure  from  train- 
ing devices  which  can  support  the  first  and  second  stages  of  learning 
before  switching  to  the  more  expensive  simulators  required  to  support 
later  learning  stages.  It  might  be  true  that  more  sophisticated, 
higher-fidelity  trainers  have  the  capability  of  teaching  what  might  be 
taught  by  less  advanced  trainers.  However,  time  thus  spent  would  be 
taken  away  from  the  time  during  which  advanced  students  could  be  per- 
fecting skills  on  the  more  advanced  trainers  (47).  "A  preferred  ap- 
proach (tc  maintenance  training)  would  be  to  use  the  simplest  material 
which  meets  the  training  objectives  and  to  select  more  complex  mater- 
ials only  when  training  objectives  cannot  be  met  efficiently  by  the 
less  costly  material" (27). 

One  often  encounters  the  belief  that  to  insure  transfer  of  train- 
ing the  instructional  settings  should  be  similar  to  the  job  situation. 

In  fact,  this  is  implied  in  AFM  50-2  (2).  Many  persons,  especially  in- 
structors, believe  that  if  you  want  to  maximize  transfer  of  training 
to  the  job,  one  of  the  better  ways  to  accomplish  this  is  to  design 
training  equipment  that  is  "as  realistic"  as  possible  (56).  It  is 
difficult  to  identify  the  factual  basis,  if  any,  for  this  belief.  In 
the  first  place,  actual  equipment  trainers  are  more  apt  to  be  used  as 
classroom  demonstrators  than  as  trainers  for  the  development  of  per- 
formance skills.  Other  media  such  as  low-cost  mock-ups  and  films  would 
serve  equally  well  as  demonstrators,  although  their  motivational  value 
might  be  less.  The  primary  objective  of  many  maintenance  courses  is  to 
train  people  to  perform  at  an  apprentice  level.  At  this  level  tasks 
can  be  performed  accurately  but  in  a rather  slow,  uncoordinated,  "un- 
skillful" manner.  Many  persons,  instructors  particularly,  believe  that 
real  equipment  is  required  to  teach  skills  at  this  level.  That  belief 
is  erroneous.  Low-cost  mock-ups  and  simulators  may  often  be  more 
appropriate. 

The  official  intent  of  most  maintenance  courses  is  to  prepare  stu- 
dents so  that,  with  additional  on-the-job  practice  and  training,  they 
can  master  their  job.  For  these  courses  the  course  performance  stan- 
dards are  considerably  lower  than  those  required  on  the  job.  Moreover, 
some  maintenance  courses  concentrate  on  teaching  theory  and  do  not  even 
attempt  to  teach  performance  skills.  Still  other  courses,  especially 
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those  based  on  some  version  of  the  task  oriented  training  concept,  have 
been  so  reduced  in  length  that  they  can  only  provide  students  with  a 
brief  introduction  to  the  equipment(s)  they  will  maintain  in  the  field. 

With  reference  to  Table  I,  such  courses  cover  maintenance  skills  and 
knowledges  at  only  the  first  or  second  stage  of  learning.  Courses 
taught  at  these  levels  have  little  if  any  valid  need  for  actual  equip- 
ment trainers  or  even  for  simulators.  Certainly  they  do  not  need 
simulators  which  can  be  used  to  teach  the  skills  of  troubleshooting. 

Before  alternatives  to  the  foregoing  approaches  to  maintenance 
training  are  developed,  it  would  seem  judicious  to  reexamine  course 
standards.  Should  it  be  the  goal  of  a 3-level  maintenance  course  to 
prepare  students  so  that,  with  minimal  additional  Field  Training  De- 
tachment (FTD)  training  and  on-the-job  training  (OJT),  they  can  master 
their  job?  If  the  answer  to  this  question  is  "yes,"  then  the  standards  for 
3-level  courses  should  be  raised.  This  in  turn  would  make  it  easier  to 
justify  the  use  in  these  courses  of  simulators  and  even  actual  equipment 
trainers. 

Assuming  that  the  objective  of  a resident  maintenance  course  is  to 
provide  graduates  who  have  reasonable  skills,  it  then  becomes  appro- 
priate to  develop  a media  mix  of  training  aids,  part-task  trainers  and 
simulators  which  will  prepare  students  so  that  they  can  rapidly  refine 
their  skills  when  they  have  access  to  real  equipment.  Whether  this 
access  should  be  provided  during  resident  training,  FTD  training  or  OJT 
should  be  decided  in  terms  of  the  difficulty,  expense  and  consequence 
of  training  recent  course  graduates  in  the  field  via  the  use  of  opera- 
tional equipment  or  dedicated  actual  equipment  trainers.  For  some  main- 
tenance specialties  it  might  be  less  expensive  to  provide  both  simula- 
tors and  actual  equipment  trainers  for  resident  training  so  as  to  reduce 
the  requirement  for  field  training  on  operational  equipment.  This  ap- 
proach would  take  maximum  advantage  of  transfer  of  training  principles 
and  the  advantages  associated  with  employment  of  simulators  for  train- 
ing. This  approach  is  based  also  on  a realistic  view  of  how  recent 
maintenance  course  graduates  are  used  in  the  field--they  are  used  as 
aides  to  senior  repairmen  until  they  have  demonstrated  an  ability  to 
work,  under  general  supervision,  on  specific  types  of  equipment. 
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DESIGN  AND  SPECIFICATION  OF  SIMULATION  REQUIREMENTS 


Current  Problem  Areas 


Many  authors  share  with  Spangenberg  (65)  the  view  that  "no  predic- 
tive body  of  knowledge  is  available  which  will  insure  the  adequate  de- 
sign of  a simulator  for  effective  training."  Both  Eggemeier  (19)  and 
Cream  (12)  cite  the  need  for  a systematic  methodology  for  matching  train- 
ing requirements  with  training  device  features,  and  it  is  the  opinion 
of  Bryan  and  Regan  (8)  that  "at  the  present  time  most  major  training 
device  design  situations  and  problems  must  be  treated  individually  since 
there  are  few  rules  for  designers." 

Many  decisions  must  be  made  when  designing  and  writing  specifica- 
tions for  a simulator.  The  physical  and  functional  fidelity  level  at 
which  various  equipment  features  will  be  simulated  must  be  decided. 
Decisions  must  be  made  about  certain  physical  features  of  the  simulator; 
how  rugged  it  will  be,  for  example.  Decisions  must  be  made  about  the 
instructional  features  to  be  incorporated  into  the  simulator,  and  who 
will  have  the  final  say  with  respect  to  these  features.  Before  final 
decisions  are  made  trade-off  analyses  must  be  made  to  determine  if  the 
decisions  are  made,  trade-off  analyses  must  be  made  to  determine  if  the 
tions.  These  and  other  issues  must  be  resolved  on  the  basis  of  general 
guidelines  only  since  no  specific  rules  exist.  Especially  troublesome 
to  make  are  decisions  about  fidelity. 

The  Fidelity  Issue 

Much  of  the  acquisition  and  support  costs  of  training  devices  can 
be  related  to  the  fidelity,  especially  the  physical  fidelity,  of  the 
device--high  fidelity  training  devices  usually  cost  more  than  low  fidel- 
ity devices.  For  this  reason  the  Air  Force  is  very  concerned  with 
identifying  those  training  situations  and  conditions  where  low  fidelity 
devices  can  be  effectively  employed. 

Three  types  of  fidelity  have  been  identified  as  being  relevant  to 
training  devices.  They  are:  physical  or  equipment  fidelity;  function- 
al or  environmental  fidelity;  and  psychological  fidelity.  Physical  or 
equipment  fidelity  refers  to  the  extent  to  which  training  equipments 
duplicate  the  appearance  and  feel  of  their  real  equipment  counterparts 
(56).  Functional  or  environmental  fidelity  refers  to  the  extent  to 
which  training  equipments  duplicate  stimuli  which  are  present  in  the 
operational  environment  and  provide  an  opportunity  for  responding  real- 
istically to  these  stimuli.  Psychological  fidelity  refers  to  the  extent 
to  which  trainees  perceive  the  training  equipment  "as  being  a duplicate 
of  the  operational  equipment  and  the  task  situation."  There  is  wide- 
spread agreement  that  to  be  effective  a training  device  should  possess  a 
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high  degree  of  functional  fidelity.  There  is  considerable  disagreement 
over  the  amount  of  physical  or  equipment  fidelity  a training  device 
should  possess. 

Numerous  studies  have  demonstrated  that  training  devices  of  low  or 
medium  physical  fidelity  can  be  effective.  Such  devices  had  been  es- 
pecially effective  for  teaching  procedures  (41,  67,  69).  In  many  of 
these  studies  the  training  device  simulated  front  panel  controls  and 
displays,  and  allowed  for  students  to  practice  making  responses  to  these 
controls,  A not  unusual  finding  has  been  that  inexpensive  mock-ups  were 
superior  to  expensive  mock-ups  and  real  equipment  for  teaching  equip- 
ment start-up  or  energizing  procedures  (1,  3,  10).  In  view  of  these 
findings  it  would  seem  that  "the  answer  to  maintenance  training  lies 
not  in  how  much  but  how  little  simulation  to  use.  This  is  not  to  say 
that  fewer  simulators  should  be  used,  merely  that  low-cost,  low- 
fidelity  simulators  should  be  employed  where  they  can  be  made  effective. 
(27) 


The  Air  Force  has  sponsored  the  development  of  a number  of  trainers 
designed  primarily  to  teach  troubleshooting  logic.  The  early  versions 
of  these  devices  usually  included  a two-dimensional  display  which  pro- 
vided symptoms  and  test  point  information  in  binary  terms.  Despite 
their  obvious  low  physical  fidelity,  they  were  found  to  be  effective 
for  teaching  fault  location  procedures  (68). 

A more  sophisticated  version  of  this  type  of  trainer,  the  EC- II/ 
EC-3,  has  been  effectively  employed  to  teach  nomenclature,  operating 
procedures,  and  malfunction  location  at  the  organizational  maintenance 
level  (39,  64,  65,  71). 

There  now  seems  to  be  no  question  but  that  training  devices  which 
possess  a reasonable  level  of  functional  validity  but  which  have  a low 
physical  fidelity  can  be  very  effective.  Despite  this  evidence,  most 
maintenance  training  devices  still  possess  an  overabundance  of  physical 
realism,  often  to  the  detriment  of  their  effectiveness.  In  fact,  in 
most  maintenance  courses  today  actual  equipment  trainers  are  extensively 
used.  Generally  they  are  used  as  classroom  demonstrators  not  as  skill 
developers.  Moreover,  they  are  not  very  effective  for  developing  skills, 
at  least  not  at  the  first  and  second  stages  of  learning. 

During  the  first  stage  of  learning  both  physical  and  functional 
fidelity  can  be  rather  low.  However,  it  appears  that  as  students  pro- 
gress through  the  second  and  third  stages  of  learning,  the  functional 
fidelity  of  supporting  training  devices  should  increase  more  rapidly 
than  the  physical  fidelity  of  those  devices  (33).  It  appears  also  that 
a point  will  be  reached  beyond  which  increases  in  either  functional  or 
physical  fidelity  will  not  increase  training  effectiveness. 
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Miller  (47)  has  postulated  that  as  the  fidelity  of  a training  de- 
vice increases,  the  amount  of  transfer  of  training  to  operational 
equipment  also  will  increase,  but  at  some  point  will  level  off  despite 
increases  in  the  fidelity  of  the  training  environment.  The  relation- 
ship postulated  by  Miller  (Figure  2)  further  suggests  that  at  some 
pint  the  additional  amount  of  transfer  of  training  obtained  from  addi- 
tional training  device  fidelity  becomes  much  less  cost-effective. 


VERY 

EXPENSIVE 


INEXPENSIVE 


Figure  2 Interrelationship  Between  Cost,  Fidelity  of 
Simulation,  ana  Training  Effectiveness 
(Modified  from  Miller,  1954). 


Eventually,  students  must  acquire  the  ability  to  perform  efficient- 
ly on  operational  equipment.  It  might  be  argued  therefore  that,  at 
some  time  during  their  formal  resident  training,  students  should  be 
provided  with  the  opportunity  to  practice  on  either  actual  equipment 
trainers  or  on  simulators  which  possess  a very  high  degree  of  physical 
and  functional  fidelity.  This  argument  is  valid  to  the  extent  that 
resident  training  must  prepare  students  to  perform  skillfully  in  the  job 
setting  without  benefit  of  additional  field  training.  This  seldom  is 
a requirement  of  resident  courses.  Therefore,  resident  maintenance 
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courses  seldom  have  high  performance  skill  objectives.  Rather,  they 
tend  to  be  theory  and  knowledge  oriented,  assuming  that  skills  will  be 
acquired  in  the  field.  Because  of  this,  during  resident  training  it 
often  is  sufficient  to  demonstrate  equipment  operation  and  maintenance 
actions  as  opposed  to  teaching  students  how  to  operate  equipment  and 
perform  maintenance  tasks. 

There  may  be  a few  advanced  (5-  and  7-level)  maintenance  courses 
which  have  terminal  objectives  identical  to  job  performance  require- 
ments. The  need  for  such  requirements  must  be  determined  on  the  basis 
of  a job  analysis  and  an  analysis  of  the  potential  for  on-the-job 
training  and  practice.  When  the  requirement  does  exist,  there  will  also 
exist  a requirement  for  the  use  of  actual  equipment  trainers  and  possi- 
bly very  high  fidelity  simulators.  More  commonly,  the  objectives  of  an 
initial  (3-level)  resident  maintenance  course  will  be  to  prepare  stu- 
dents so  that  via  FTD  training  and  on-the-job  training  and  practice 
they  can  perform  at  the  journeyman  level  after  some  months  on  the  job. 
For  such  courses  a high  degree  of  transfer  of  training  between  resident 
training  and  the  job  is  not  expected.  Therefore,  there  is  only  a mini- 
mum requirement  for  using  actual  equipment  trainers  or  high  physical- 
fidelity  simulators  during  training. 

During  the  early  stages  of  learning,  the  use  of  high  fidelity  simu- 
lators may  interfere  with  effective  learning  (3).  While  learning 
nomenclature  and  the  location  of  front  panel  controls  and  displays,  or 
when  learning  the  relationship  between  controls  and  equipment  opera- 
tion, there  is  little  need  to  represent  all  the  other  competing  stimuli 
and  response  options  which  may  exist  in  the  operational  environment. 
Their  presence  would  only  confuse  the  students  and  slow  up  their  learn- 
ing process. 

Learning  to  perform  portions  of  a complicated  procedure  on  a 
specially  constructed  part-task  trainer  has  been  found  to  be  an  effec- 
tive way  to  learn  complicated  procedures.  For  example,  it  would  be 
both  impractical  and  ineffective  to  teach  the  use  of  an  oscilloscope 
while  teaching  also  the  procedures  for  locating  a particular  class  of 
malfunctions. 

Reilly  (56)  has  provided  a list  of  21  characteristics  for  describ- 
ing training  devices  (Table  III).  Each  characteristic  describes  a 
feature  which  might  be  incorporated  into  a simulator.  For  example,  to 
the  extent  possible,  training  devices  should  have  structural  or  physical 
fidelity,  functional  fidelity,  and  provide  for  the  learning  of  both 
motor  skills  and  cognitive  skills.  In  addition,  it  is  advantageous  to 
develop  simulators  which  are  student-proof,  allow  students  to  interact 
with  the  simulator  and  to  receive  feedback,  can  be  used  extensively 
throughout  a course,  are  relatively  inexpensive  to  purchase  and  main- 
tain, and  so  on.  Obviously,  no  one  training  device  can  possess  all  21 
characteristics  in  favorable  amounts.  Actual  equipment  trainers  possess 
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Table  III  List  of  Training  Device  Characteristics 
{Adapted  from  Reilly) 
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high  amounts  of  structural  and  functional  fidelity  and  allow  students 
to  practice  motor  skills.  They  are  not,  however,  studentproof , easily 
modified  or  updated,  easily  maintained,  etc.  It  cannot  be  overstated 
that  characteristics  which  make  equipment  effective  in  the  operational 
setting  often  are  those  which  make  it  ineffective  as  a trainer. 

Desired  Physical  and  Instructional  Features 

Well-designed  training  equipment  intentionally  deviates  from  real- 
ity (8).  To  promote  learning,  it  should  possess  certain  features  not 
possessed  by  its  real  equipment  counterpart.  To  withstand  heavy  use  by 
both  students  and  instructors  it  should  be  constructed  differently  than 
the  equipment  it  represents.  To  prolong  its  usefulness  it  should  be 
capable  of  being  modified  to  reflect  later  versions  of  the  equipment  it 
simulates. 

Some  of  the  instructional  featu.es  which  should  be  incorporated  in- 
to a simulator  include  the  capability  to  provide  prompts  and  cues;  to 
provide  feedback  to  the  learner;  to  allow  the  instructor  to  vary  the 
difficulty  level  of  problems;  and  to  allow  for  variations  in  the  train- 
ing sequence  in  order  to  adapt  training  to  the  present  state  of  the 
trainee  (8).  In  addition,  Kinkade  and  Wheaton  (33)  mention  the  capa- 
bility to  teach  specific  tasks,  and  to  allow  practice  on  difficult 
components  of  a lengthy  procedure,  as  useful  instructional  features. 

Some  special  features  which  seem  desirable  (8)  are  (a)  the  ability  to 
set  the  device  so  that  it  will  demonstrate  both  proper  and  improper 
operations  and  common  mistakes;  (b)  the  capability  to  record  student 
performance  on  video  for  playback  purposes;  (c)  the  capability  to 
provide  for  practice  and  drill  on  difficult  segments  of  a procedure; 
and  (d)  the  capability  to  exercise  individual  positions  in  a team 
trainer.  Which  of  these  many  features  should  be  incorporated  into  any 
specific  trainer  is  a matter  of  opinion;  most  decisions  are  made  on  the 
basis  of  educated  guesses. 

An  especially  troublesome  decision  area,  at  least  to  the  authors 
of  this  report,  concerns  whether  or  not  to  incorporate  into  a simulator 
the  capability  to  simulate  many  malfunctions.  Some  people  feel  that 
there  is  little,  if  any,  need  to  increase  the  troubleshooting  ability  of 
course  graduates.  This  opinion  seems  based  on  the  fact  that  present  3- 
level  course  standards  seldom  require  students  to  be  proficient  trouble- 
shooters at  the  end  of  the  course.  In  theory,  this  skill  is  acquired 
in  the  field.  In  practice  this  policy  is  likely  to  produce  graduates 
with  so  few  practical  skills  that  field  personnel  dislike  trying  to 
teach  them  on  the  job. 

Another  version  of  this  opinion  has  been  expressed  by  Cream  (12) 
who  has  suggested  that  training  device  designers  should  not  incorporate 
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into  their  device  a large  malfunction  insertion  capability.  Rather,  he 
proposed  that  designers  first  determine  the  amount  of  course  hours  avail- 
able for  troubleshooting  practice.  Since  these  hours  will  not  be  enough 
to  cover  a large  number  of  malfunctions,  he  recommends  that  designers 
identify  critical  malfunctions  and  simulate  only  those.  Our  objection 
to  this  position  is  based  on  the  belief  that  maintenance  course  gradu- 
ates should  have  some  reasonable  ability  to  troubleshoot  when  they  ar- 
rive at  the  field  so  that  job  incumbents  will  regard  them  as  worthy  at 
least  to  be  helpers.  We  agree  with  Malone  that  "troubleshooting  train- 
ing must  include  as  many  as  possible  critical  and/or  frequent  failure 
modes."  (37) 

The  authors  feel  that  current  courses  have  slighted  troubleshoot- 
ing training  because  of  difficulty  in  inserting  malfunctions  into  real 
equipment.  Under  current  conditions  it  often  is  too  much  trouble  to 
teach  practical  troubleshooting  skills.  This  need  not  and  should  net 
be  the  case.  Moreover,  we  feel  that  the  advantage  of  simulators  should 
be  viewed  with  respect  to  the  total  training  system,  both  the  school- 
house  and  the  field.  We  think  it  is  important  and  cost-effective  to 
raise  formal  course  standards  when  the  raised  standards  can  be  met  at 
no  or  only  slight  increases  in  training  costs.  This  may  not  provide  a 
payoff  to  ATC  Technical  Training  Centers,  but  it  might  considerably 
reduce  the  cost  of  FTD  training.  In  addition,  it  should  increase  the 
operational  capability  of  field  units,  and  perhaps  save  money  which 
otherwise  would  have  to  be  spent  inefficiently  on  on-the-job  training. 

In  addition  to  instructional  features,  the  well-designed  simulator 
should  possess  a number  of  physical  characteristics.  Of  course,  it 
should  represent  its  equipment  counterpart  in  a manner  that  will  pro- 
mote the  acquisition  of  job-oriented  training  objectives.  Moreover, 
special  efforts  should  be  made  to  functionally  represent  those  tasks 
which  will  be  taught  on  the  trainer.  In  addition,  the  device  should 
provide  easy  access;  be  rugged  or  student-proof;  be  reliable  and 
easily  maintained;  be  as  simple  as  possible;  and  be  able  to  with- 
stand unusual  use  patterns  (33).  To  this  list  Bryan  and  Regan  (8) 
would  add  that  the  device  should  be  easy  to  reprogram;  be  constructed 
in  a module  or  building-block  design;  and  when  possible,  be  designed 
for  multiple  usage  by  different  types  of  trainees.  As  discussed  under 
the  section  on  fidelity,  Reilly  (56)  has  provided  a list  of  21  char- 
acteristics which  can  be  employed  to  both  describe  and  review  the  de- 
sign requirements  for  simulators.  Some  of  these  characteristics  are 
mutually  exclusive,  and  there  are  no  hard-and-fast  rules  for  deciding 
the  degree  to  which  they  should  exist  in  a training  device. 

Procedures  for  Selecting  Requirements 

Despite  the  lack  of  rules,  decisions  eventually  must  be  made  about 
the  design  and  specifications  for  simulators.  Bryan  and  Regan  provide 
a useful  discussion  on  the  use  of  trade-off  analyses.  These  will  help 
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one  identify  the  costs  and  benefits  of  a training  device  which  will  sup- 
port various  training  requirements,  simulate  various  equipment  functions, 
and  have  certain  instructional  features.  Their  discussion  offers  a few 
guidelines  but  no  prescriptions.  They  also  point  out  that  the  trade-off 
analysis  should  not  stop  after  comparing  training  device  options.  Ra- 
ther, non-hardware  alternatives  also  should  be  considered.  For  example, 
perhaps  a training  requirement  can  be  more  cost-effectively  handled  by 
low-cost  aids  or  mock-ups.  Perhaps  training  should  be  conducted  on  the 
job.  Perhaps  no  training  is  required  at  all;  instead,  well-constructed 
job  aids  might  be  used  to  guide  performance  on  the  job  (22,  50). 

Shriver  et  al . (61)  used  the  exemplary  model  approach  to  identify 
the  requirements  for  a simulator  of  the  NIKE  HIPAR  system.  Based  on 
prior  research,  they  already  had  identified  the  functional  characteris- 
tics of  the  operating  and  maintenance  mock-ups  they  felt  were  needed 
for  training  on  new  radar  systems.  Further,  they  ascertained  that  the 
data  required  to  develop  these  devices  and  their  supporting  documents 
existed  in  manufacturers'  manuals  used  for  "Key  Personnel"  training. 

Also,  they  found  that  prototype  equipment  was  available  for  study  at 
test  and  evaluation  sites.  Most  importantly,  they  found  that  these 
information  sources  were  available  at  least  one  and  one-half  years  be- 
fore student  training  had  to  begin.  This  gave  them  ample  time  to 
develop  a system  simulator,  supporting  maintenance  documents  and 
classroom  lesson  plans.  This  illustration  serves  as  a reminder  that 
the  information  needed  to  make  decisions  about  the  purchase  of  simu- 
lators can  be  different  from,  and  must  be  available  earlier  than, 
the  more  detailed  information  upon  which  the  engineering  features  of 
the  simulator  are  based.  To  anticipate  a later  discussion,  we  feel 
that  the  exemplary  model  approach  does  allow  one  to  make  simulator 
purchasing  decisions  early  in  the  new  equipment  development  cycle. 
Development  of  specifications  for  the  functional  characteristics  of 
the  simulator  then  can  await  the  production  of  an  advanced  prototype 
of  the  new  equipment. 

While  reviewing  the  use  of  part-task  trainers  for  maintenance, 
Reilly  (56)  noted  that  new  trainers  such  as  the  EC-3,  and  trainers 
based  on  Honeywell's  Automated  Electronic  Maintenance  Trainer  (AEMT) 
concept,  have  been  designed  to  provide  a multi-system  application 
capability.  For  example,  the  EC-3  trainer  can  simulate  equipment  at 
all  levels  of  maintenance.1  Given  the  capability  to  rapidly  change 


^Besides  the  Educational  Equipment  Corporation  and  Honeywell,  there  are 
many  other  companies  which  have  a capability  to  produce  either  or  both 
two-  and  three-dimensional  maintenance  trainers.  These  companies  in- 
clude Autonetics,  AVCO,  Boeing,  General  Electric,  Parsons  and  Sylvania. 
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from  one  level  of  simulation  to  another,  there  is  no  need  to  incorpor- 
ate both  levels  into  the  same  simulation  package.  Also,  there  is  no 
need  to  try  to  simulate  two  or  more  subsystems  with  the  same  simulation 
package.  It  sometimes  is  cheaper  to  use  two  simulators  to  meet  all  simu- 
lation requirements  than  it  is  to  develop  a single  simulator  which  meets 
all  requirements.  In  a similar  vein,  the  EC-3  uses  different  panels  and 
programs  to  simulate  different  subsystems. 

Sometimes  equipment  consists  of  many  parts  which  are  virtually 
identical  to  one  another.  When  this  situation  exists,  one  or  two 
parts  can  be  functionally  simulated  and  photos,  mock-ups  or  dummy  com- 
ponents used  to  simulate  the  remainder  (44).  This  technique  was  follow- 
ed in  order  to  lower  the  cost  of  the  Air  Force's  6883  Converter/Flight 
Controls  Test  Station  Simulator. 

There  seems  to  be  some  disagreement  over  the  relative  role  various 
groups  should  play  in  the  selection  of  simulator  characteristics.  Most 
authors  agree  that  there  should  be  an  interaction  between  the  user  (in- 
structors who  use  the  simulator),  simulation  engineers  and  training 
psychologists.  Cream  (12)  proposes  that  instructors  make  many  of  the 
final  decisions  on  the  basis  of  information  provided  by  engineers  and 
psychologists.  In  part,  this  woul  1 be  an  effort  to  ensure  appropriate 
instructor  usage  of  the  simulate  :s  well  as  to  lessen  their  negative 
feelings  about  the  device.  The  authors  of  this  report  recognize  the 
problem  of  obtaining  instructor  cooperation  but  would  prefer  to  search 
for  solutions  elsewhere.  We  would  prefer  to  obtain  inputs  from  instruc- 
tors but  then  let  training  psychologists  and  engineers  knowledgeable 
about  simulation  techniques  make  the  final  decisions  on  the  basis  of 
trade-off  analysis.  Training  psychologists  are  more  apt  to  know  about 
the  cost  and  benefits  of  a variety  of  features  which  can  be  incorporated 
into  simulators.  Moreover,  they  are  more  apt  to  consider  low-cost  train- 
ing alternatives  as  well  as  non-training  alternatives  such  as  more 
"human  engineering"  of  equipment.  Engineers  expert  on  simulation  tech- 
niques are  more  apt  to  know  the  feasible  techniques  for  simulating 
various  equipment  features  and  maintenance  tasks,  and  the  approximate 
cost  of  various  simulation  approaches. 

In  a previous  section  on  the  determination  of  simulation  require- 
ments this  report  briefly  discussed  the  possibility  of  copying  exem- 
plary training  devices  employed  to  teach  the  maintenance  of  other 
similar  weapons  systems.  We  find  this  an  attractive  approach  and  sus- 
pect that  it  is  followed  rather  extensively,  the  flaw  in  its  applica- 
tion being  that  the  simulator  copied  probably  are  not  especially 
exemplary. 
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USER  ACCEPTANCE  OF  MAINTENANCE  SIMULATORS 


Research  Findings  and  Problem  Areas 


During  recent  years  there  has  been  an  increased  realization  that 
the  effectiveness  of  training  devices  depends  on  how  they  are  used  and 
accepted  by  instructors  and  students  (40).  Instructors,  however,  are 
the  more  important  of  these  two  user  groups  because  they  are  in  a posi- 
tion to  defeat  the  purposes  of  training  devices  and  to  govern  the  atti- 
tudes of  students  toward  them. 

According  to  anecdotal  evidence,  training  devices  are  continually 
misused,  and  usually  underused,  by  instructors.  Instructors  ignore 
many  of  the  capabilities  built  into  flight  simulators.  They  used  main- 
tenance simulators  as  classroom  demonstrators,  showing  students  how  to 
locate  malfunctions  instead  of  allowing  them  to  practice  a skill. 

When  viewed  from  a different  perspective,  instructors  berate  simulator 
designers  for  developing  over-elaborate  training  equipment  whose  many 
"nice  to  have"  features  have  no  practical  value  in  the  classroom  or 
training  lab.  Both  parties  to  this  dispute  can  cite  arguments  in  their 
favor  (8). 

The  few  research  findings  that  relate  to  this  problem  area  indicate 
that  instructors  favor  the  use  of  maintenance  simulators  so  long  as 
their  use  is  under  the  control  of  instructors,  and  so  long  as  simulators 
are  not  used  to  reduce  the  inventory  of  actual  equipment  trainers.  Mc- 
Guirk  (39)  reported  that  the  EC-II  troubleshooting  trainer  was  well 
received  by  instructors  and  students  and  subsequently  was  incorporated 
into  a Weapons  Control  Systems'  Mechanic  Course  concerned  with  the 
AN/APQ-126  Radar  System.  Spangenberg  (65)  found  that  national  guards- 
men were  impressed  with  the  effectiveness  of  the  EC-II  trainer  but  felt 
that  it  was  better  overall  to  use  actual  equipment  for  training.  Biers- 
ner  (6)  found  that  Navy  instructors  and  trainees  favored  use  of  the 
EC-II  for  troubleshooting  training  over  use  of  actual  equipment  trainers. 
The  Automated  Electronic  Maintenance  Trainer  (AEMT)  concept  has  been  used 
to  simulate  the  AN/ALQ-100  airborne  electronic  warfare  transceiver  and 
Navy  instructors  found  the  simulator  to  be  acceptable  so  long  as  it  was 
not  usud  to  replace  actual  equipment  trainers  (16). 

Modrick  (48)  admitted  that  trainers  based  on  the  AEMT  concept 
attempt  to  provide  a high  fidelity  simulation  of  how  equipment  "looks" 
and  "feels"  iri  an  effort  to  safeguard  against  negative  attitudes  of  in- 
structors and  other  opponents  of  simulation.  He  noted  also  that  Navy 
reviewers  agreed  that  the  AEMT  appeared  capable  of  teaching  trouble- 
shooting procedures  but  felt  that  a "trainer  should  be  an  aid  to  the 
instructor  rather  than  a replacement  for  him." 
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Even  flight  simulator  advocates  continue  to  experience  user  accept- 
ance resistance.  According  to  Killian  (30)  the  extensive  use  of  simu- 
lators requires  "a  step-by-step  program  of  proving  and  selling  each  new 
concept  to  the  pilots  who  are  involved,  the  management  who  must  support 
the  developments,  the  approval  and  authorizing  agencies,  and  the  instruc- 
tor groups  who  can  make  or  break  a program." 

Much  of  the  resistance  to  simulation  is  based  on  the  belief  that 
high  physical  fidelity  of  a training  device  is  necessary.  Neither  in- 
structors nor  students  can  readily  understand  how  training  on  simulators, 
and  especially  low  fidelity  simulators  or  mock-ups,  can  prepare  students 
to  work  on  actual  equipment.  As  mentioned  earlier  in  this  report,  per- 
haps the  use  of  simulators  has  been  oversold  or  mis-sold  in  the  sense 
that  proponents  of  them  sometimes  imply  that  al 1 training  can  best  be 
conducted  using  simulators  and  other  training  devices.  We  do  not  sub- 
scribe to  this  view,  and  neither  do  many  simulator  advocates;  instruc- 
tors should  be  informed  of  this.  In  our  view,  a cost-effective  main- 
tenance program  is  most  apt  to  be  one  that  employs  a mix  of  training 
aids,  simulators,  and  actual  equipment  trainers;  one  type  of  training 
aid  or  device  would  not  be  eliminated  in  favor  of  other  types.  Rather, 
low-cost,  low  fidelity  aids  and  devices  would  be  employed  where  possi- 
ble so  as  to  reduce,  but  seldom  eliminate,  the  number  of  more  expensive 
simulators  and  AETs  required  to  support  training. 

Techniques  for  Obtaining  Instructor  Acceptance 

Bryan  and  Regan  (8)  have  reviewed  the  features  which  should  be 
possessed  by  simulators,  and  these  features,  when  present,  should  make 
simulators  more  acceptable  to  instructors.  The  features  include:  high 
reliability,  ease  of  programming  and  reprogramming,  ease  of  access  and 
use.  Also,  the  simulator  should  employ  an  instructor  station  which 
allows  the  instructor  to  be  a "good"  instructor.  That  is,  the  station 
should  provide  information  about  the  range  of  problems  available  to  and 
for  use  by  the  instructor;  it  should  be  easy  to  load  programs  into  the 
simulator;  it  should  be  easy  to  reconfigure  the  trainer  to  meet  the 
needs  of  individual  students  or  a particular  section  of  a course;  and 
information  about  student  performance  should  be  provided.  To  this  list 
it  should  be  added  that  the  simulator  should  not  be  over-elaborate, 
should  not  overload  the  instructor,  should  not  collect  useless  data,  and 
should  provide  feedback  to  the  student.  Of  course,  the  simulator  should 
have  a high  degree  of  functional  fidelity  and  a medium  amount  of  physical 
fidelity.  Unfortunately,  the  definitions  for  "high"  and  "medium"  fidel- 
ity are  not  clear. 

Cream  (12)  is  of  the  opinion  that  instructors  should  have  the  final 
say  with  respect  to  the  design  of  a simulator,  and  should  be  responsible 
for  preparing  the  plan  for  integrating  the  device  into  the  training  pro- 
gram. He  feels  that  this  is  the  best  way  to  assure  that  the  device  will 
be  acceptable  and  be  used.  He  notes  also  the  need  for  communication  be- 
tween instructors,  simulator  designers  and  training  psychologists  during 
the  design  stage  of  the  simulator.  These  are  the  persons  who  should  be 
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involved  in  the  trade-off  analysis  which  should  be  conducted  before 
the  functional  specifications  for  a simulator  are  finalized.  Bryan  and 
Regan  (8)  apparently  feel  that  less  reliance  should  be  placed  on  in- 
structor decisions.  However,  they  do  emphasize  the  need  for  communi- 
cation between  simulator  designers  and  instructors,  and  recommend 
that  instructors  have  some  say  regarding  design  parameters.  They 
recommend  also  that  a joint  committee  composed  of  instructors, 
simulator  cesigners  and  instructional  psychologists  be  established 
for  the  purpose  of  developing  instructional  material  and  the  plans  for 
incorporating  the  device  into  a course. 

There  is  some  disagreement  regarding  who  should  prepare  the  utiliza- 
tion guide  for  a simulator,  although  there  is  agreement  that  one  should 
be  prepared.  Cream  (12)  feels  that  it  should  be  prepared  by  instruc- 
tors, while  Bryan  and  Regan  (8)  feel  that  simulator  designers,  with  the 
cooperation  of  users,  should  be  responsible  for  the  utilization  guide. 
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THE  COST-EFFECTIVENESS  OF  MAINTENANCE  SIMULATORS 


The  Potential  and  Reality  of  Simulators 

So  far  in  this  report  we  have  reviewed  numerous  studies,  many  of 
which  claim  that  simulators  and/or  low-cost  training  devices  are  more 
cost-effective  for  training  than  are  actual  equipment  trainers.  Siecko 
(64)  noted  that  the  EC-II  usually  cost  only  a fraction  of  the  cost  of 
the  real  equipment  it  represented.  Spangenberg  (65)  reported  on  how 
effective  and  relatively  inexpensive  was  the  EC-II  trainer.  French  (23) 
noted  that  the  MAC-1  and  MAC-2  trainers  were  much  less  costly  than  the 
systems  they  represented,  and  were  more  effective.  The  Automated  Elec- 
tronic Maintenance  Trainer  (AEMT)  is  rather  expensive,  yet  one  report 
noted  (15)  that  it  was  less  than  half  the  cost  of  the  system  ( AN/ALQ- 
100)  which  it  simulated. 

Another  group  of  authors  have  reported  on  the  cost-effectiveness 
of  training  aids  and  mock-ups.  Torkelson  (67)  doubted  if  the  use  of 
three-dimensional  mock-ups  could  be  justified  when  wall  charts  or  trans- 
parencies served  equally  well  at  much  less  cost.  Vris  (69)  concluded 
that  cutaway  charts  and  transparencies  were  economical  and  effective 
substitutes  for  more  expensive  three-dimensional  cutaways.  Grimsley 
(25)  varied  the  appearance  and  functional  fidelity  of  a panel  used  to 
train  soldiers  to  operate  the  Section  Control  Indicator  Console  of  the 
Ni’;e  Hercules  guided  missile  system  and  found  all  three  versions  of 
the  panel  to  be  equally  effective.  As  noted  already,  Cox  (10)  could 
M'»d  no  d-'  "erence  in  the  effectiveness  between  various  training  de- 
vices whi  ...  differed  in  cost. 

The  evidence  seems  convincing:  simulators  are  more  cost-effective 
for  teaching  malfunction  location  techniques  than  are  actual  equipment 
trainers;  low-cost,  low  fidelity  mock-ups  for  teaching  nomenclature, 
parts  location  and  procedures  may  be  more  cost-effective  than  more  ex- 
pensive three-dimensional  aids,  animated  cutaways,  or  real  equipment 
trainers.  Surely,  the  time  has  now  arrived  for  the  wholesale  adoption 
of  simulators  and  low-cost  training  devices,  or  has  it?  Has  the  use 
of  maintenance  sir-  ’ators  resulted  in  the  use  of  lesser  numbers  of  actual 
equipment  t’diners.  Has  their  use  resulted  in  a decrease  in  course 
training  time?  has  any  course  been  able  to  reduce  its  requirement  for 
instructors  because  of  the  use  of  simulators?  These  are  the  basic  ways 
to  reduce  course  costs,  yet  there  is  no  published  evidence  to  indicate 
that  these  costs  have  been  lowered  by  the  use  of  maintenance  simulators. 
The  literature  does  tain  estimates  of  the  amount  of  money  which  might 
be  saved  through  t’  ,c>e  of  simulators  (48)  and  many  authors  have  specu- 

lated that  it  shouiu  be  fairly  easy  to  obtain  cost  savings  through  the 
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use  of  simulators.  However,  the  literature  does  not  contain  reported 
examples  of  such  savings  in  cost.  It  is  difficult  and  time-consuming 
to  evaluate  the  effects  of  training  devices  and  perhaps  it  is  for  this 
reason  that  cost  advantages  associated  with  the  use  of  simulators  have 
not  been  reported.  In  addition,  the  extensive  use  of  maintenance  simu- 
lators is  a very  recent  phenomenon  and  their  effectiveness  is  still 
undergoing  evaluation.  We  can  only  hope  that  those  involved  with  cur- 
rent and  future  maintenance  simulator  studies  will  publish  their 
results  so  that  the  presumed  cost-effectiveness  of  maintenance 
simulators  can  be  adequately  documented. 

Moving  on  to  the  topic  of  effectiveness,  what  evidence  is  there 
that  maintenance  simulators  have  resulted  in  greater  end-of-course 
student  proficiency  which  in  turn  has  transferred  to  the  field  in  some 
noticeable  fashion?  The  authors  of  this  report  know  of  no  studies  on 
maintenance  simulators  which  have  addressed  the  transfer  of  training 
issue  in  this  fasion,  although  anecdotal  evidence  exists  that  students 
trained  with  simulators  can  perform  acceptably  in  the  field.  An  an 
example,  in  the  early  1960s  a course  developed  under  the  direction  of 
Shriver,  Fink,  and  Trexler  (61)  was  forced  to  train  students  to  maintain 
the  NIKE  HIPAR  radar  system  before  the  real  equipment  showed  up  at  the 
training  site  (a  not  unusual  occurrence).  A series  of  specially  de- 
signed simulators,  based  on  a detailed  task  analysis,  was  used  instead 
of  real  equipment  as  training  devices.  The  first  few  classes  of  stu- 
dents were  trained  with  these  devices  with  no  known  damaging  impact  on 
field  operations.  However,  no  data  were  collected  to  identify  what, 
if  any,  problems  course  graduates  did  experience  in  the  field.  Even- 
tually the  HIPAR  equipment  showed  up  at  the  school.  The  course  then 
became  quite  conventional,  employing  both  real  equipment  and  the 
specially  designed  simulators  for  training.  In  so  doing  the  course 
became  more  costly,  possibly  more  effective,  and  in  all  probability, 
less  cost-effective. 

The  typical  study  of  maintenance  simulators  compares  students 
trained  on  a troubleshooting  simulator  with  students  trained  to  trouble- 
shoot on  actual  equipment.  Training  on  the  simulator  is  supplemented 
with  nomenclature  and  parts  location  training.  Both  groups  receive 
special  training  on  the  use  of  test  equipment.  Both  student  groups 
then  are  tested  on  some  type  of  troubleshooting  test  which  usually  em- 
phasizes the  application  of  troubleshooting  logic,  a skill  which  can  be 
taught  well  using  a maintenance  simulator.  Invariably  students  trained 
on  the  maintenance  simulator  perform  better,  or  at  least  equally  as  well, 
on  the  test  as  do  students  trained  on  actual  equipment  trainers. 

After  the  usual  simulator  experiment  is  terminated,  the  troubleshoot 
ing  simulator  may  be  incorporated  into  the  regular  course  and  the  course 
Plan  of  Instruction  (POI)  adjusted  so  that  students  now  troubleshoot  on 
both  the  simulator  and  on  actual  equipment  trainers.  The  course  is  never 
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shortened  because  of  this,  and  little  attention  is  paid  to  the  increased 
proficiency  which  the  students  may  now  have.  It  is  doubtful  if  field 
supervisors  are  even  aware  that  students  are  now  being  trained  slightly 
differently.  From  the  standpoint  of  school  policy,  the  course  now  is 
producing  students  which  exceed  course  requirements,  but  this  is  an 
acceptable  situation  so  long  as  it  doesn't  cost  much.  In  fact,  if 
training  personnel  like  the  new  simulator,  they  may  order  more  of  them. 
Eventually  they  may  end  up  with  a course  which  is  considerably  more 
costly  in  terms  of  training  equipment  requirements,  and  which  over- 
trains its  students,  at  least  with  respect  to  current  course  standards. 

Despite  claims  for  their  cost-effectiveness,  the  published  facts 
indicate  that  the  advantages  to  be  gained  through  the  use  of  mainten- 
ance simulators  remain  as  a potential;  the  potential  gains  have  yet 
to.  be  translated  into  practical  impacts.  The  challenge  for  the  future, 
then,  is  not  one  of  showing  that  maintenance  simulators  are  potentially 
cost-effective.  Rather,  it  is  to  identify  the  training  conditions  and 
assumptions  which  must  be  changed  so  that  this  potential  can  be  real- 
ized, and  to  actually  effect  these  changes. 

Realizing  the  Potential  of  Simulators 

If  simulators  are  cost-effective,  what  must  be  done  to  demonstrate 
this  in  a practical  fashion?  Let's  consider  the  possibilities.  Con- 
sider a course  where  course  standards  are  not  to  be  changed  and  where 
simulators  are  to  be  used  to  teach  troubleshooting  skills.  The  in- 
creased effectiveness  of  the  simulator  may  allow  one  to  shorten  the 
length  of  the  course.  Perhaps  40  hours  of  POI  time  are  devoted  to 
troubleshooting  training.  Using  a simulator  this  may  be  reduced  by 
one  or  two  days  with  no  loss  in  effectiveness,  not  much  of  a cost  sav- 
ing unless  the  course  trains  large  numbers  of  students. 

Perhaps  the  course  employs  two  or  more  actual  equipment  trainers 
during  the  troubleshooting  exercise.  Using  simulators,  one  or  more  of 
these  AETs  may  not  be  needed.  Perhaps  they  can  be  completely  dropped 
from  the  training  equipment  inventory,  but  probably  not.  Instructors 
will  still  want  AETs  to  serve  as  demonstrators  to  both  students  and 
visitors.  To  a visitor  the  presence  of  actual  equipment  trainers  is 
more  impressive  than  the  mere  presence  of  low-cost  simulators.  AETs 
give  the  impression  that  students  actually  are  taught  to  maintain  them. 
The  facts  may  be  quite  different,  but  the  visitor  does  not  stay  long 
enough  to  learn  of  this. 

Perhaps  by  using  simulators  the  instructor  requirements  for  main- 
tenance courses  can  be  decreased.  This  might  be  difficult  to  accomplish 
because  troubleshooting  occupies  only  a small  percentage  of  time  in  the 
typical  maintenance  course,  and  instructors  are  needed  for  the  "infor- 
mation pumping"  aspects  of  the  course.  However,  if  the  simulator  is 
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designed  to  take  over  the  giving  of  instruction,  as  in  the  CAI  mode, 
for  example,  the  requirement  for  instructors  could  decrease. 

So  far  then  it  seems  that  the  use  of  simulators  instead  of  real 
equipment  for  troubleshooting  will  effect  cost  savings  if  an  actual 
equipment  trainer  can  be  dropped  from  the  course,  or,  if  the  simulator 
takes  on  some  of  the  instructional  load  so  that  instructor  require- 
ments can  be  reduced. 

Consider  now  a course  where  both  course  costs  and  standards  can  be 
changed,  where  it  is  permissible  and  even  judged  worthwhile  to  produce 
graduates  who  have  an  increased  troubleshooting  proficiency.  To  obtain 
this  increased  proficiency  usually  will  require  an  increase  in  training 
equipment  costs.  The  impact  of  the  increased  proficiency  may  be  that 
course  graduates  are  now  able  to  more  readily  assume  their  duties  in 
the  field.  The  operational  readiness  of  tho  units  to  which  they  are 
assigned  actually  may  be  increased,  but  this  would  be  extremely  difficult 
to  prove.  It  might  be  less  difficult  to  demonstrate  that  lesser  amounts 
of  on-the-job  training  now  are  required  to  bring  course  graduates  up  to 
a required  proficiency.  This  may  result  in  a cost  savings  to  field  units 
but  not  to  the  training  center. 

For  courses  already  in  existence,  the  addition  of  simulators  will 
increase  training  center  costs  unless  they  can  eliminate  certain  num- 
bers of  actual  equipment  trainers,  or  can  shorten  training.  Some 
courses  only  employ  one  actual  equipment  trainer  so  there  seems  little 
chance  of  eliminating  the  use  of  that  trainer.  Those  courses  which 
employ  two  or  more  AETs  may  be  able  to  eliminate  one  or  more  of  them, 
and  effect  a considerable  savings. 

Taking  another  approach,  the  standards  of  a course  can  be  examined 
more  thoroughly,  especially  with  respect  to  their  job  relevance.  Is  it 
the  objective  of  a maintenance  course  to  teach  job-relevant  skills  and 
knowledges?  If  so,  perhaps  the  course  should  concentrate  more  on  teach- 
ing students  to  do  something  as  opposed  to  teaching  them  about  something. 
Maintenance  simulators  are  most  effective  when  used  to  teach  and  to 
provide  practice  on  job-oriented  skills;  they  will  be  cost-effective  to 
the  extent  that  course  objectives  emphasize  the  acquisition  of  job  per- 
formance skills.  Despite  the  emphasis  on  the  ISD  approach  to  course 
development,  most  maintenance  courses  still  are  oriented  toward  the 
acquisition  of  theoretical  knowledges  as  opposed  to  job-relevant  skills. 

To  summarize  the  arguments  presented  here,  the  authors  are  of  the 
opinion  that  published  research  to  date  has  only  suggested  that  main- 
tenance simulators  are  potentially  cost-effective.  Furthermore,  we 
feel  that  the  potential  benefits  of  simulators  cannot  be  achieved  unless 
certain  fundamental  changes  are  made  to  maintenance  courses.  These 
changes  involve  one  or  more  of  the  following: 
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changing  course  standards  to  emphasize  the  acquisition 
of  proficiency  as  opposed  to  knowledges. 

raising  course  standards  to  the  point  where  it  will 
take  a mixture  of  simulators  and  AETs  to  achieve  them. 

requiring  that  course  lengths  be  reduced  with  no  loss 
in  course  graduate  proficiency. 

specifying  that  for  any  particular  course  only  one  copy 
of  the  actual  equipment  can  be  used  for  both  operator  and 
maintenance  training,  and  that  simulators  and  other  train 
ing  devices  must  be  used  instead  of  additional  actual 
equipment  trainers. 
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SUGGESTIONS  FOR  FUTURE  RESEARCH 


The  research  evidence  has  demonstrated  that  "many  procedures  and 
troubleshooting  tasks  can  be  effectively  simulated  with  relatively  low 
levels  of  fidelity"  (25,  26).  Further,  this  evidence  indicates  that  "as 
training  requirements  move  from  the  'nuts  and  bolts'  toward  the  more 
cognitive  aspects  of  understanding  electrical,  electronic  and  hydraulic 
systems  and  the  acquisition  of  diagnostic  and  troubleshooting  skills, 
programmable  computer-operated  devices  become  more  appropriate  than 
actual  equipment  (37)."  Regardless  of  the  evidence,  however,  the  use 
of  simulators  for  maintenance  training  has  not  received  widespread 
support,  and  many  questions  about  the  use  of  and  effectiveness  of  main- 
tenance simulators  still  remain.  The  paragraphs  that  follow  describe 
much  of  the  research  which  should,  according  to  recent  articles  and 
reports,  be  performed  with  reference  to  maintenance  simulators  and 
other  media  for  maintenance  training. 

Because  of  the  current  pace  of  maintenance  simulator  development 
and  implementation,  we  feel  that  it  will  be  difficult  for  research  to 
stay  ahead  of  implementation.  We  suspect  that  for  some  time  to  come 
researchers  will  be  reacting  to  hardware  developments  rather  than 
identifying  the  directions  these  developments  should  take.  Despite 
this,  further  research  must  at  least  prepare  the  right  questions  to  be 
asked  of  implemented  products,  especially  as  they  pertain  to  product 
evaluation.  Such  questions  should  improve  the  next  iteration  of  main- 
tenance training  devices  and  point  future  simulator  R&D  in  more  cost- 
effective  directions. 

Evaluating  the  Cost-Effectiveness  of  Simulators 

Those  who  advocate  the  use  of  simulators  for  maintenance  training 
believe  that  simulators  can  both  increase  the  effectiveness  of  train- 
ing and  decrease  costs.  Maybe  so,  but  convincing  evidence  to  support 
these  claims  has  still  to  be  offered.  Simulators  are  effective  during 
training;  at  least  they  are  more  effective  than  are  actual  equipment 
trainers  for  teaching  troubleshooting  skills.  However,  there  is  little 
evidence  regarding  the  degree  to  which  simulators  promote  transfer  of 
training  to  the  field  (24,  39).  More  evidence  is  needed  in  this  area. 

In  particular,  we  need  more  information  about  what  can  best  be  taught 
by  the  use  of  simulators,  procedures  or  cognitive  troubleshooting  skills, 
with  "best  be  taught"  evaluated  in  terms  of  transfer  of  training  to  the 
field. 

Simulators  usually  cost  less  than  the  equipment  they  represent. 

It  follows  then  that  training  costs  should  be  reduced  when  simulators 
are  used  in  place  of  real  equipment.  The  flaw  in  this  logic  is  that  it 
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is  based  on  the  potential  of  simulators.  So  far  few  maintenance 
courses  have  substituted  simulators  for  actual  equipment  trainers. 

Instead,  simulators  have  been  added  to  the  course  equipment  inventory. 
Further  studies,  either  on  new  systems  or  on  extant  ones,  must  actually 
do  without  real  equipment  and  demonstrate  that  the  impact  is  not 
detrimental.  To  quote  from  King  (31),  "What  seems  to  be  needed  are 
adequate  demonstrations,  to  the  user  communities,  of  those  cost  reduc- 
tions and  training  benefits  that  are  likely  to  accrue  from  the  use  of 
these  new  technologies.  The  time  is  right  for  putting  together  the 
best  of  each  of  the  different  approaches  (Tech  Pubs,  simulations,  CAI) 
into  one  or  more  advanced  systems.  The  objective  of  each  would  be  to 
provide  the  user  community  with  the  experience  and  data  they  need  to 
commit  to  an  investment  in  the  newer  methods." 

Before  the  cost-effectiveness  of  simulators  can  be  demonstrated, 
the  indices  of  training  costs  and  effectiveness  of  training  must  be  re- 
examined, and  the  methods  for  conducting  evaluative  studies  reassessed. 

As  an  example,  training  effectiveness  can  be  defined  in  terms  of  with- 
in or  end-of-course  proficiency,  proficiency  during  early  weeks  on  the 
job,  or  proficiency  after  some  months  on  the  job.  So  far  transfer  of 
training  research  has  demonstrated  that  simulators  can  increase  pro- 
ficiency during  and  at  the  end  of  resident  training.  A variety  of 
research  also  has  demonstrated  that  major  variations  in  resident  train- 
ing cannot  be  correlated  with  job  proficiency  after  a year  or  more  in 
the  field.  It  still  remains  to  be  shown  conclusively  that  transfer  of 
training  affects  performance  during  the  first  weeks  and  months  on  the 
job.  If  transfer  of  training  does  occur,  it  should  have  its  most  im- 
portant and  noticeable  impact  during  these  weeks.  There  is  a need 
therefore  for  studies  which  examine  the  relationship  between  resident 
training,  especially  that  supported  by  simulators,  and  on-the-job  pro- 
ficiency during  the  first  one  to  three  months  on  the  job.  In  addition, 
there  is  a need  for  more  research  in  the  field  of  trainer  evaluation, 
especially  the  evaluation  of  training  effects  resulting  from  the  manner 
in  which  a trainer  is  used  versus  those  effects  related  to  the  trainer 
itself  (53). 

Proficiency  during  early  weeks  on  the  job  can  show  itself  in  a var- 
iety of  ways;  these  indices  should  be  identified  and  incorporated  into 
evaluation  studies.  As  examples,  has  the  pattern  of  job  assignment 
been  changed?  Has  FTD  training  or  on-the-job  training  been  shortened?  Has 
the  time  required  to  certify  recent  graduates  at  the  "5"  level  been  de- 
creased? These  are  some  of  the  data  that  should  be  collected  in  future 
transfer  of  training  studies. 

Those  persons  conducting  evaluative  studies  of  simulators  should 
be  alert  to  the  possibility  that  supervisors  will  not  be  aware  that 
recent  course  graduates  may  now  have  increased  capabilities  and  can  be 
given  more  difficult  assignments  during  their  early  months  on  the  job. 
Assuming  this  to  be  the  case,  it  may  be  necessary  to  educate  super- 
visors regarding  these  new  capabilities  and  to  suggest  to  them  that  they 
employ  recent  graduates  differently  than  they  did  formerly. 
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With  reference  to  training  costs,  it  seems  quite  possible  that 
simulators  may  lead  to  only  slight  cost  reductions  for  school  training 
but  may  provide  the  potential  for  substantially  reducing  the  cost  of 
FTD  and  on-the-job  training.  As  an  illustration,  it  may  be  more  cost- 
effective  to  employ  simulators  to  reduce  the  need  for  actual  equipment 
training  but  not  to  reduce  course  length.  Probably  this  would  lead  to 
the  overtraining  of  students,  at  least  with  respect  to  current  course 
standards.  In  turn,  this  might  necessitate  raising  the  standards.  The 
practical  implication  of  the  foregoing  approach  is  that  simulators 
might  reduce  somewhat  the  cost  of  resident  training,  but  that  the  major 
cost  impact  might  be  felt  elsewhere.  It  is  our  view  that  the  cost- 
effectiveness  of  simulators  should  be  evaluated  in  terms  o.  the  total 
impact  on  the  personnel  selection  and  training  system,  and  on  the  logis- 
tics system  instead  of  merely  the  impact  on  a particular  course  or 
Technical  Training  Center. 

Comparing  Simulators 

Despite  appearance,  there  is  little  evidence  that  the  application 
of  simulator  technology  has  increased  the  proficiency  of  maintenance 
course  graduates.  New  simulators  have  been  evaluated  by  comparing 
students  trained  on  them  with  persons  trained  in  the  classroom  or  on 
real  equipment.  Montemerlo  (49),  Spangenberg  (65)  and  others  have 
severely  criticized  this  research  paradigm,  pointing  out  that  this 
type  of  research  does  not  increase  our  understanding  of  the  conditions 
for  which  simulators  best  can  be  employed.  Furthermore,  research  com- 
paring simulators  with  actual  equipment  trainers  has  implied  that 
simulators  can  be  substituted  for  actual  equipment  trainers  in  toto. 
There  is  no  evidence  that  this  is  possible.  It  is  worth  noting,  how- 
ever, chat  it  is  useful  to  conduct  a few  simulator  studies  solely  for 
the  purpose  of  demonstrating  that  simulators  can  be  effective  trainers. 
Such  public  relations  efforts  can  pave  the  way  for  further  studies 
which  employ  more  suitable  research  paradigms. 

In  general,  studies  on  the  effectiveness  of  maintenance  simulators 
have  compared  persons  trained  to  troubleshoot  on  a simulator  with  per- 
sons who  have  not  been  trained  to  troubleshoot  but  who  perhaps  have 
talked  their  way  through  a few  troubleshooting  exercises  in  the  class- 
room. With  rare  exceptions,  the  "new  trainer"  always  wins.  Students 
trained  on  the  EC-3  or  the  AEMT  have  not  been  compared  with  students 
taught  on  the  EC- 1 1 . Even  worse,  they  have  not  been  compared  with 
students  taught  on  the  MAC-1  malfunction  and  circuitry  trainer  or  any 
of  the  other  trainers  developed  years  ago.  For  all  we  know  we  may  have 
gained  nothing,  at  least  with  respect  to  course  graduate  proficiency, 
despite  recent  advances  in  simulator  technology.  Research  is  required 
to  examine  this  possibility. 

It  may  be  difficult  to  win  support  tor  comparative  research  on 
simulators  because  no  precedent  lias  been  established  for  conducting 


this  type  of  research.  Moreover,  there  is  some  question  as  to  whether 
or  not  ATC  schools  should  have  the  responsibility  for  producing  gradu- 
ates with  practical  troubleshooting  capabilities.  Some  feel  that  this 
should  be  the  responsibility  of  field  training.  On  the  other  hand,  a 
capability  that  actually  exists  can  create  a need  in  ways  that  a con- 
ceptual capability  cannot.  Now  that  there  are  simulators  that  can  be 

used  effectively  to  teach  troubleshooting  skills  during  resident  train- 
ing, more  and  more  persons  may  agree  that  this  should  be  allowed  even 
though  it  necessitates  an  upgrading  of  course  standards. 

Supposedly  recently  developed  trainers  and  simulators  have  a num- 
ber of  favorable  characteristics  not  possessed  by  simulators  of  the 
fifties  and  early  sixties.  Presumably  they  can  cover  a wider  variety 
of  malfunctions  than  trainers  of  the  past.  It  is  claimed  that  they  are 

less  subject  to  obsolescence  because  they  can  be  modified  more  readily. 

New  simulators  possess  somewhat  higher  degrees  of  both  functional  and 
physical  fidelity  and  therefore  should  be  more  effective.  Those  are 
some  of  the  arguments  offered  in  favor  of  recent  trainers  and  simula- 
tors. The  validity  of  these  arguments  remains  unknown  and  should  be 
the  subject  of  further  research. 

Deter' ;ninq  Training  Requirements 

A number  of  authors  have  proposed  research  programs  which  might 
lead  to  a better  identification  of  training  device  requirements.  As 
mentioned  earlier,  Bryan  and  Regan  (8)  feel  that  more  basic  research 
should  be  performed  on  job  and  task  analysis  techniques,  especially 
on  what  additional  information  might  oe  gathered  to  support  later  de- 
cisions  about  training  requirements  and  media  support  requirements. 

They  also  recommend  more  research  on  how  to  teach  various  tasks  which 
are  common  to  many  maintenance  job  positions. 

Both  Cream  et  al.  and  Eggemeier  (13,  19)  have  suggested  that  more  re 
search  is  needed  cn  "short-term"  methods  for  identifying  training  ob- 
jectives and  requirements.  G.  G.  Miller  (44)  has  discussed  one  tech- 
nique for  selecting  from  a pool  of  requirements  those  which  should  be 
simulated.  More  research  is  needed  on  the  effectiveness  of  this  and 
alternate  methods.  Modrick  (48)  has  discussed  the  need  for  more  re- 
search on  the  methodology  for  converting  engineering  data  into  training 
and  training  device  requirements.  The  foregoing  research  problems 
are  especially  acute  with  respect  to  new  weapons  systems;  there  is  a 
reed  to  develop  improved  techniques  for  identifying  training  and  train- 
ing device  requirements  early  in  the  weapons  systems  development  cycle 
so  that  decisions  can  be  made  about  the  types  of  trainers  to  purchase. 

In  a recent  paper,  Klein  (34)  noted  that  ISD  procedures  were  not 
developed  to  provide  data  ir.  support  of  the  design  of  simulators.  In 
particular  he  felt  that  "ISD  procedures  do  not  adequately  describe 
complex  motor  coordination  tasks,  nor  tasks  requiring  complex  perceptual- 
cognitive  decisions.  Questions  of  necessary  fidelity,  criteria  for 
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selecting  part-task  vs.  full  mission  simulators,  or  types  of  instruc- 
tional features  are  not  addressed."  It  seems  probable  that  the  initial 
steps  In  the  ISD  process  can  be  modified  of  elaborated  to  provide  in- 
formation needed  to  design  simulators.  This  possibility  should  be 
examined. 

A practical  approach  to  some  of  the  above  problems  has  been  pro- 
posed by  Cream  et  al.  (13)  and  others,  and  is  based  on  the  observation  that 
new  weapons  systems  seldom  are  completely  new  but  rather  are  improve- 
ments to  present  systems.  Cream  proposed  that,  when  identifying  train- 
ing and  training  device  requirements,  one  should  look  at  how  training  is 
conducted  on  similar  equipments  and  tasks.  Extensions  to  this  approach 
will  be  discussed  later  on. 

Determining  Simulation  Requirements 

Principles  for  determining  training  content  and  simulation  require- 
ments for  various  learning  objectives  exist  only  in  rudimentary  form, 
and  need  much  more  development.  We  need  better  data  regarding  what 
simulators  and  actual  equipment  trainers  can  best  be  used  for,  and  at 
what  stage  in  the  training  process  each  should  be  used  separately  or 
in  various  mixes  (49,  52,  56). 

Research  relating  to  the  degree  to  which  simulators  should  physi- 
cally and  functionally  represent  equipment  should  receive  continued 
support,  since  it  is  arguments  about  this  issue  tnat  often  lead  to  the 
non-acceptance  of  low-cost  training  devices  and  simulators  (33). 

Just  who  should  be  involved  in  the  determination  of  simulation 
requirements,  and  especially  which  group  should  make  the  final  decision, 
is  a matter  for  further  study.  Most  authors  agree  that  simulation 
engineers,  instructors,  and  training  psychologists  should  be  involved, 
but  there  is  disagreement  regarding  the  roles  they  should  play  and  the 
decisions  they  should  make. 

Numerous  authors  have  discussed  the  need  for  uetermiring  simula- 
tion requirements  on  the  basis  of  a trade-off  analysis.  The  literature, 
together  with  military  documents,  is  replete  with  guidelines,  usually 
quite  general,  for  how  to  conduct  such  an  analysis  and  to  make  decisions 
regarding  the  design  of  simulators.  In  addition,  there  ere  many  mili- 
tary persons  and  civilians  in  the  Air  Force  who  have  participated  in 
this  decision-making  activity.  It  would  seem  useful  to  collect  and 
codify  this  expertise  and  these  guidelines,  and  to  incorporate  them 
into  a manual  for  the  design  of  maintenance  simulators.  We  admit  that 
this  would  riot  lead  to  simple  formulas  which  could  be  followed  by 
the  uninformed.  However,  it  should  at  least  collate  the  stata-of-the- 
art  and  help  spot  more  precisely  the  gaps  in  it. 
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Developing  Exemplary  Simulators 


One  way  to  identify  training  device  requirements  for  new  weapons 
systems  is  to  examine  how  training  is  conducted  on  similar  extant 
systems.  An  extension  of  this  approach  is  to  identify  tasks  common 
to  many  systems  and  then  to  identify  exemplary  traininq  devices  al- 
ready in  existence  which  can  be  used  to  teach  those  tasks.  Comple- 
menting this  approach  is  one  which  attempts  to  develop  exemplary 
training  devices  for  common  maintenance  tasks  or  common  equipments 
or  equipment  subsystems. 

The  approach  to  simulator  development  currently  followed  by  the 
Air  Force  involves  the  identification  of  common,  costly  equipments 
which  might  be  simulated  for  the  purpose  of  maintenance  traininq.  The 
development  of  a simulator  for  the  6883  Converter/Flight  Control  Test 
Station  for  the  F-111D  aircraft  is  an  example  of  this  approach.  In 
addition,  the  authors  of  this  report  currently  are  conducting  a study 
wh  ich  will  attempt  to  identify  other  common  Air  Force  equipments  or 
weapons  subsystems  which  might  be  simulated  cost-effectively  during 
training. 

For  selected  subsystems  which  appear  in  many  equipments,  exem- 
plary materials  for  teaching  maintenance  should  be  developed.  One 
should  select  subsystems  where  heavy  use  currently  is  made  of  actual 
equipment  trainers.  The  goal  of  the  research  effort  would  be  to  re- 
duce, but  not  entirely  eliminate,  the  use  oi  actual  equipment  trainers, 
and  in  their  stead  develop  one  or  more  simulators  and  perhaps  lower 
cost  training  devices  to  prepare  students  for  the  simulators. 

The  approach  just  described  should  receive  continued  support. 

But,  in  addition,  for  each  simulator  developed  a more  detailed  docu- 
mentation should  be  made  of  such  activities  as  the  identification  of 
training  requirements,  the  selection  of  training  requirements  to 
simulate,  the  trade-off  analysis  process,  and  so  on.  This  documenta- 
tion should  make  it  easier  to  develop  future  simulators,  and  should 
provide  the  information  which  eventually  should  be  incorporated  into 
official  Air  Force  maintenance  simulator  development  documents  and 
regulations. 

Various  authors  (48)  have  suggested  that  generic  training  modules 
should  be  developed  to  cover  tasks  that  occur  across  many  equipment 
systems.  Expanding  on  this  notion,  many  Air  Force  weapons  systems  and 
equipments  have  one  or  more  of  the  following  subsystem- -electronic, 
electrical,  hydraulic,  mechanical  or  optical.  It  would  seem  worth- 
while to  develop  model  approaches  and  simulators  for  teaching  these 
subsystems.  Moreover,  it  would  seem  useful  to  investigate  the  possi- 
bility of  developing  exemplary  approaches  to  teach  various  maintenance 
activities— removal  and  replacement  of  parts,  inspection  and  preventive 
maintenance,  troubleshooting— on  common  equipment  subsystems.  The  goal 
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would  be  to  develop  exemplary  packages  or  modules  which  could  be 
copied  by  new  weapons  systems.  This  would  allow  the  developers  of  a 
training  program  for  a new  weapons  system  to  specify,  at  least  in 
general  terms,  the  type  of  trainers  they  wanted  to  purchase  and  the 
probable  trainer  costs. 

Developing  Exemplary  Mixes  of  Maintenance  Training  Media 

It  has  been  suggested  by  many  that  training  aids,  simulators  and 
actual  equipment  trainers  are  most  effective  at  specific  stages  in  the 
learning  process,  and  this  supposition  has  been  adopted  in  this  report. 
There  is  considerable  need,  however,  for  more  data  on  what  maintenance 
skills  and  knowledges  various  aids,  trainers  and  simulators  can  support 
most  effectively,  and  at  what  point(s)  in  the  training  process  each 
should  be  used  (49,  56).  This  information  could  be  developed  as  part 
of  the  proposed  development  of  exemplary  approaches  to  teach  various 
maintenance  activities  for  various  types  of  equipment.  In  addition  to 
investigating  what  types  of  simulators  are  most  appropriate,  it  would 
seem  appropriate  to  develop  exemplary  media  mixes  for  various  systems 
and  subsystems  and  their  supporting  equipment. 

In  a recent  report  Montemerlo  (49)  contended  that  studies  which 
compare  the  effectiveness  of  simulators  with  that  of  actual  equipment 
trainers  are  based  on  the  wrong  research  approach.  Rather,  he  feels 
that  research  efforts  should  be  concentrated  on  the  identification  of 
the  mix  of  training  media-training  aids,  mock-ups,  simulators,  and 
actual  equipment  trainers— which  are  more  cost-effective  for  mainten- 
ance training.  Montemerlo  also  pointed  out  that  with  one  exception, 
the  Apollo  moon  landing  project,  simulators  always  have  been  used  in 
conjunction  with  actual  equipment  trainers,  and  that  this  will  be  the 
case  for  many  future  years.  "Given  that  simulators  are  an  adjunct  to 
actual  equipment  in  training,  the  question  of  whether  one  is  better 
than  the  other  becomes  spurious,  whereas  the  problem  of  allocating 
training  objectives  between  the  two  rises  to  a position  of  pre- 
eminence." He  concluded  his  report  by  stating  that  the  goal  of  future 
research  should  be  to  determine  the  types  of  training  objectives  and 
conditions  which  are  conducive  to  each  type  of  training  media,  and 
that  research  should  be  designed  to  determine:  the  unique  contribution 
each  type  of  training  device  can  make;  the  ways  each  can  complement 
the  other;  the  training  methods  which  amplify  the  advantages  of  each; 
and  the  types  of  training  objectives  most  conducive  to  each. 

In  addition  to  supporting  the  use  of  a variety  of  training  devices, 
the  present  authors  feel  that  the  effectiveness  of  simulators  can  be 
increased  by  more  closely  intertwining  the  acquisition  of  maintenance 
knowledges  and  skills.  We  feel  that  knowledges  should  be  learned  in 
the  context  of  their  application;  this  is  the  essence  of  the  "functional 
context"  approach  to  learning. 
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In  the  traditional  maintenance  course  knowledges  are  taught  early 
in  the  course  and  skills  are  acquired  later  on.  Considerable  time  is 
devoted  to  providing  instruction  about  equipment  but  little  time  is 
devoted  to  the  practical  application  of  what  was  learned  previously. 

The  problems  with  this  approach  are  many.  In  particular,  the  practical 
application  portion  of  the  course  often  is  slighted.  As  a result 
students  are  required  to  remember  instructional  material  for  a consid- 
erable time  before  applying  it  to  real  equipment  in  the  field.  This 
temporal  gap  be\  aen  instruction  and  its  application  places  too  much  of 
a burden  on  long-term  memory;  by  the  time  course  graduates  arrive  in 
h'  **  Id,  they  have  forgotten  much  of  the  application-oriented  know- 

I ..  s they  once  had  learned.  There  is  no  apparent  advantage  to  storing 
roal  content  in  long-term  memory.  This  practice  appears  to  be  an 
artifact  of  educational  practices  which  do  not  prepare  people  for 
specific  equipment-oriented  jobs. 

An  alternative  approach  combines  knowledge  learning  with  its 
practical  application.  This  should  be  a more  effective  approach  since 
it  reduces  the  requirement  for  long-term  memory  for  instructional 
material  to  a short-term  one--the  short  gap  between  instruction  and  its 
practical  application.  In  addition,  this  approach  may  lessen  the  amount 
of  nice-to-know  information  offered  in  most  maintenance  courses.  More- 
over, memory  will  be  improved  with  the  mix  of  verbal  and  spatial  pro- 
cessing. 

CAI  programs  which  employ  computer-graphics  terminals  offer  one 
means  for  teaching  related  knowledges  and  skills  in  close  temporal 
sequence.  Recent  efforts  to  incorporate  CAI  capabilities  into  mainte- 
nance simulators  may  provide  another  technique  for  keeping  the  instruc- 
tional and  application  process  intertwined. 

Bryan  and  Regan  (8)  believe  that  inadequate  support  has  been  pro- 
vided for  basic  research  on  simulators,  and  feel  that  an  effort  should 
be  made  to  identify  more  precisely  the  simulator  characteristics  re- 
quired to  support  the  learning  of  various  maintenance  tasks  as  various 
stages  in  the  learning.  They  feel  also  that  more  reserach  should  be 
conducted  which  compares  the  cost-effectiveness  of  simulators  with  non- 
hardware alternatives  such  as  low-cost  media,  job  performance  aids, 
ana  on-the-job  training. 

Obtaining  User  Acceptance 

There  is  continuing  evidence  that  instructors  do  not  believe  in  the 
effectiveness  of  simulators  and  do  not  properly  utilize  them  even  when 
they  are  supplied  in  adequate  numbers.  Concrete  suggestions  as  to  what 
to  do  about  this  situation  seldom  appear  in  the  literature,  probably  be- 
cause the  problem  is  not  one  that  is  subjected  easily  to  research. 
Rather,  it  is  more  of  a public  relations  problem. 
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Some  of  the  techniques  which  might  be  employed  to  obtain  user 
acceptance  should  be  described  more  fully,  and  instances  where  one  or 
more  of  these  techniques  have  proven  successful  should  continue  to  be 
reported  in  the  literature.  Mackie,  et  al.  (36)  have  made  a good  start 
on  this.  They  identified  six  factors  which  lead  to  the  acceptance  or 
rejection  of  training  devices.  These  simulation  factors  are:  specific 
trainer  features,  characteristics  of  the  users,  characteristics  of  the 
instruction,  pattern  of  use,  and  the  manner  in  which  the  device  is  in- 
troduced to  the  user.  Some  of  the  practical  ways  to  obtain  better 
acceptance  of  simulators  by  instructors  have  been  described  by  Cream  (12). 
They  include  efforts  to:  assure  continuity  of  the  user;  assure  user 
participation  in  test  and  acceptance  of  device;  carefully  plan  delivery 
of  the  device  to  the  training  setting,  to  include  the  development  of  a 
utilization  plan;  and  provide  adequate  operator  and  maintenance  docu- 
mentation at  time  of  delivery.  Continued  studies  are  required  to 
determine  how  these  and  other  factors  affect  simulator  acceptance. 
Eventually  a handbook  of  case  studies  might  be  published.  In  the  mean- 
while, advocates  of  simulators  must  continue  to  discuss  the  advantages 
of  simulators  with  all  those  who  will  listen.  In  particular,  an  effort 
should  be  made  to  win  the  support  of  senior  Air  Force  officers  for  the 
use  of  maintenance  simulators;  it  is  the  attitude  of  these  officers  that 
establishes  the  "party  line"  throughout  the  training  establishment. 

The  means  by  which  groups  of  designers,  psychologists  and  instruc- 
tors should  jointly  develop  the  functional  specifications  for  simulators 
merits  further  study  and  documentation.  At  the  very  least,  instructor 
heeds  and  biases  should  be  considered  when  designing  simulators.  Senior 
instructors  should  play  a major  role  in  the  trade-off  analysis  process 
which  establishes  the  final  functional  requirements  for  simulators.  Also, 
senior  instructors  should  be  recruited  to  publicize  the  advantages  of 
simulators.  Finally,  it  would  seem  mandatory  that  a utilization  guide 
be  prepared  for  each  major  simulator,  and  that  instructors  should  have 
considerable  involvement  with  its  preparation. 

Impact  of  Automatic  Test  Equipment  (ATE) 

It  has  been  predicted  that  beginning  in  the  1980s  considerable 
diagnostic  and  fault  isolation  of  line  replaceable  units  will  be  per- 
formed by  automatic  test  equipment.  The  Navy  is  well  along  in  its 
effort  to  implement  this  maintenance  philosophy  on  board  ship,  especially 
submarines  (55).  The  more  advanced  automatic  test  equipments  employ 
computer-driven  software  packages  to  both  diagnose  and  check  out  LRUs, 
and  to  isolate  malfunctions  to  a replaceable  module.  However,  because 
of  the  wide  mix  of  equipment  which  must  be  maintained  by  many  technicians, 
it  is  still  not  possible  to  develop  accurate  diagnostic  software  for 
many  of  them  or  for  portions  of  them.  Therefore,  in  many  instances  at 
present  the  fault-locating  portion  of  Fault  Detection  and  Isolation 
(FDI)  packages  must  be  supplemented  by  the  use  of  documented  manual 
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procedures  with  or  without  the  use  of  external  test  equipment.  In  a 
few  instances  the  entire  troubleshooting  process  must  be  based  on  im- 
provised troubleshooting  supported  by  general  procedures,  drawings  and 
references. 

Implications  stemming  from  the  extensive  use  of  automatic  test 
equipment  are  many.  First,  much  of  current  maintenance  training 
still  is  oriented  towards  teaching  the  basic  theoretical  background 
required  to  develop  "improvised"  troubleshooting  procedures.  This 
training  approach  is  based  on  an  outmoded  maintenance  philosophy,  at 
least  with  respect  to  the  use  of  automatic  test  equipment  for  organiza- 
tional and  intermediate  level  maintenance. 

Second,  as  the  maintenance  of  equipment  becomes  increasingly 
systematic  based  on  self -diagnostics,  it  will  become  increasingly  im- 
portant that  technicians  be  highly  skilled  operators  of  test  equip- 
ment, especially  its  set-up. 

Third,  it  appears  that  technician  training  will  become  "increas- 
ingly dependent  upon  systems  knowledge  and  front  panel /keyboard  mani- 
pulations to  isolate  failures  to  an  individual  module  or  a group  of 
modules."  (55)  This  implies  that  technicians  will  be  much  more  de- 
pendent on  their  knowledge  of  functional  or  signal-flow  theory  as 
opposed  to  an  understanding  of  the  equipment  in  terms  of  electronic 
theory. 

Ironically,  the  use  of  automatic  test  equipment  may  both  increase 
and  decrease  the  need  for  persons  who  can  improvise  troubleshooting 
procedures.  With  respect  to  electronic  equipment,  ATEs  eventually 
should  be  able  to  diagnose  and  locate  faults  in  95  percent  of  the 
equipment  95  percent  of  the  time.  However,  when  this  potential  is 
achieved,  the  problem  of  false  diagnosis  will  become  more  critical-- 
the  technician  will  have  to  decide  if  the  unit  under  test  contains  a 
malfunction  or  if  the  test  equipment  itself  is  malfunctioning.  King  & 
Tennyson  (32)  predict  that  such  problems  will  "trigger  more  widespread 
use  of  manual  backup  techniques  to  find  'those  five  percent  of  the 
glitches'  that  the  machine  won't  catch." 

Possible  solutions  to  the  above  problems  are  being  explored  by  the 
Navy.  These  possibilities  include  the  use  of  less  skilled  first- 
enlistment  technicians  who,  should  they  decide  to  re-enlist,  would  be 
given  extensive  technical  training.  Supplementing  this  would  be  the 
extensive  use  of  improved  maintenance  manuals  coupled  with  the  heavy 
use  of  stand-alone , compact  computer-based  simulators  which  can  be  used 
both  as  training  and  maintenance  aids. 

In  light  of  the  foregoing  discussion,  and  the  interest  of  the  Air 
Force  in  the  use  of  automatic  test  equipment,  it  would  seem  appropriate 
to  identify  more  precisely  the  job  requirements  of  technicians  who  will 
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employ  automatic  test  equipment.  Based  on  this  analysis,  the  entire 
approach  to  the  training  of  these  technicians  should  be  re-examined. 

It  seems  probable  that  the  training  approach  currently  planned  for 
these  technicians  is  inappropriate.  Moreover,  it  seems  probable  that 
these  technicians  will  need  to  be  provided  with  improved  job  perform- 
ance aids,  aids  that  can  be  used  to  identify  false  diagnoses  and  to 
handle  the  problems  which  the  test  equipment  cannot  handle.  Finally, 
there  is  some  evidence  that  the  use  of  automatic  test  equipment  will 
lead  to  the  use  of  troubleshooting  procedures  which  are  more  like  those 
now  used  for  organizational  level  troubleshootinq--reliance  on  the  use 
of  front  panel  indicators  and  a functional  knowledge  of  signal  flow. 

If  this  is  so,  then  the  use  of  flat-panel,  computer-driven  simulators 
or  computer-based  simulation  may  be  more  cost-effective  than  the  use 
of  three-dimensional  simulators  such  as  those  currently  planned  for 
the  6883  Converter/Flight  Control  Test  Station. 

Developing  Improved  Regulations  for  Maintenance  Simulators 

Better  and  more  detailed  procedures  must  be  developed  for  inte- 
grating maintenance  trainer  design  into  the  weapons  system  development 
cycle.  As  it  is  now,  decisions  about  maintenance  training  devices  often 
are  made  before  alternative  approaches  to  training  have  been  examined. 

Asa  result  actual  equipment  trainers  are  ordered  instead  of  less 
costly  and  probably  more  effective  simulators  (37). 

Currently  Kinton  is  under  contract  with  the  Technical  Training  Division 
of  the  Air  Force  Human  Resources  Laboratory  to  investigate  the  procedures 
by  which  simulators  are  obtained  for  new  weapons  systems.  What  is  obvious 
so  far  is  that  there  are  more  detailed  procedures  for  securing  flight 
simulator  than  maintenance  simulators.  Procedures  for  integrating  the 
design  and  procurement  of  maintenance  simulators  and  other  types  of 
maintenance  trainers  should  be  developed.  In  addition,  there  is  a need 
to  identify  the  points  in  the  weapons  system  development  cycle  where 
cooperation  between  maintenance  simulator  designers,  training  psychologists, 
instructors,  and  other  important  groups  should  occur.  These  points  could 
be  written  up  as  requirements  in  future  regulations.  We  can  agree  with 
Aultman  (4)  that  "if  the  proper  mix  of  training  mock-ups,  actual  hardware 
and  other  training  aids  is  to  occur,  early  interface  between  the  ATC  and 
the  operational  command  is  required.  Using  commands  must  be  involved  in 
the  training  area  from  the  very  beginning  of  new  equipment  acquisition." 
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